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ABSTRACT 

This  report  reviews  literature  relevant  to  team  training  in  complex  environments.  While 
technological  developments  allow  for  the  training  of  higher-order  cognitive  skills  in  complex 
simulated  environments,  in  the  absence  of  sound  learning  methodologies,  training  systems  may 
not  fully  achieve  their  desired  objectives.  There  are  relatively  few  attempts  in  the  literature  that 
focus  on  how  best  to  use  technology  to  support  effective  training,  and  little  research  effort  has 
involved  die  use  of  technology  in  the  development  of  effective  training  programs  for  teams  rather 
than  individuals.  The  effectiveness  of  team  training  systems,  and  specifically,  the  measures  of 
team  outcomes  and  team  processes  that  could  be  used  to  measure  team  performance  in 
distributed  training,  are  also  reviewed.  Some  areas  for  future  research  relevant  to  distributed 
team  training  are  identified. 
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Executive  Summary 


This  report  was  prepared  for  Task  Number  AIR  01/248  Distributed  Mission  Training 
Systems  for  RAAF.  The  Distributed  Mission  Training  (DMT)  task  at  DSTO  is  initially 
focussing  on  RAAF  F/A-18  operational  requirements  involving  Aircrew,  Regional 
Operator  Centre  Command  Teams,  and  Air  Defence  Ground  Environment  Tactical  Data 
System  Operators. 

This  report  reviews  literature  relevant  to  team  training  in  complex  environments.  The  first 
section  reviews  the  psychological  literature  concerning  cognitive  processes  for  teams,  with 
a  focus  on  shared  situation  awareness,  mental  models,  and  team  decision-making.  The 
main  aim  of  this  section  is  to  highlight  some  of  the  higher-order  processes  that  are 
believed  to  be  highly  developed  in  effective  teams. 

The  second  section  reviews  the  more  prominent  and  relevant  forms  of  team  training  and 
instructional  strategies.  While  the  Event-Based  Approach  to  Training  teams  in  complex 
high  technology  environments  has  been  widely  used  in  the  military  domain,  there  may 
also  be  a  role  for  other  forms  of  team  training,  such  as  team  coordination  training  and 
cross  training,  in  distributed  team  training.  The  extent  to  which  these  other  forms  of  team 
training  might  be  involved  in  distributed  training  would  depend  largely  on  the  extent  to 
which  the  critical  team  processes  are  adequately  covered  in  the  EBAT  approach. 

The  final  section  of  the  report  considers  the  measurement  of  team  performance.  Team 
outcome  measures  assess  whether  the  team  as  a  whole  was  successful  in  achieving  its 
objectives,  whereas  team  process  measures  are  required  to  determine  how  the  team  went 
about  achieving  its  objectives.  The  significant  majority  of  this  section  concerns  measures  of 
team  processes.  Measures  of  the  core  dimensions  of  teamwork  identify  the  degree  to 
which  teamwork  behaviours  are  displayed  by  the  team.  These  measurement  tools  can 
contain  generic  teamwork  behaviours,  in  the  case  of  the  Teamwork  Observation  Measure, 
or  identified  teamwork  behaviours  that  are  specific  to  a  given  scenario  for  a  given  team,  in 
the  case  of  the  Targeted  Acceptable  Responses  to  Generated  Events  or  Tasks  approach. 
While  being  more  labour  intensive,  this  latter  approach  offers  the  greatest  potential  for 
measurement  of  team  processes  for  Distributed  Mission  Training.  The  measures  of  shared 
mental  models  and  situation  awareness  are  very  much  aimed  at  the  individual  level  and 
collated  to  represent  team  values,  hence  further  effort  should  be  directed  towards 
exploring  measures  where  the  team  is  the  unit  of  analysis. 

While  there  have  been  many  studies  that  have  examined  different  aspects  of  team  training 
and  performance,  very  little  is  known  about  team  processes  and  performance  for 


distributed  teams.  Some  future  areas  of  research  are  identified  that  aim  to  increase  our 
understanding  and  measurement  of  distributed  team  processes  and  ultimately  enhance 
the  effectiveness  of  Distributed  Mission  Training. 
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1.  Introduction 

This  report  was  written  for  Task  Number  AIR 01 /248  Distributed  Mission  Training  (DMT) 
Systems  for  RAAF.  The  DMT  task  at  DSTO  is  initially  focussing  on  RAAF  F/A-18 
operational  requirements  involving  Aircrew,  Regional  Operator  Centre  Command  Teams, 
and  Air  Defence  Ground  Environment  Tactical  Data  System  Operators. 

This  report  reviews  literature  relevant  to  team  training  in  complex  environments. 
Technological  developments  allow  for  the  training  of  higher-order  cognitive  skills 
associated  with  human  performance  in  complex  simulated  environments.  Technology 
alone  however  does  not  ensure  that  effective  training  will  occur.  In  the  absence  of  sound 
learning  methodologies,  training  systems  may  not  fully  achieve  their  desired  objectives. 
There  are  relatively  few  attempts  in  the  literature  that  focus  on  how  best  to  use  technology 
to  support  effective  training.  This  latter  point  is  a  critical  issue  and  is  the  major  focus  of 
this  review.  Furthermore,  very  little  research  effort  has  involved  the  use  of  technology  in 
the  development  of  effective  training  programs  for  teams  rather  than  individuals.  Another 
key  focus  area  of  this  report  is  the  effectiveness  of  training  systems,  and  specifically, 
exploring  the  measures  of  team  outcomes  and  team  processes  that  could  be  used  to 
measure  team  performance  in  distributed  training.  In  essence  the  core  question  that  is 
being  addressed  in  this  review  is  this  -  How  should  technology  best  be  used  to  provide  an 
efficient  and  effective  environment  for  distributed  team  training? 

Before  commencing  with  the  review  of  the  literature  it  is  instructive  to  explore  the  concept 
of  DMT  in  the  Australian  context.  There  are  some  objectives  of  DMT  that  become 
apparent  when  considering  RAAF  sources  of  information.  Two  broad  objectives  are  to 
develop  an  aircrew  training  system  that  accommodates  mission  rehearsal,  and  to  develop 
an  aircrew  training  system  that  provides  effective  team  training  and  learning  of  cognitive 
skills  involved  in  complex  air  operations. 

The  first  objective  relating  to  mission  rehearsal  encompasses  the  belief  that  forces  should 
"train  the  way  we  fight'  (Layton,  2000).  Training  needs  to  prepare  die  forces  for  combat  by 
simulating  as  closely  as  possible  the  combat  environment  in  which  they  are  likely  to  find 
themselves.  Exercising  and  training  only  one  component  of  the  force,  or  training  each  part 
separate  from  the  others,  results  in  no  opportunity  for  exercising  the  interactions  and 
building  the  cohesion  that  will  be  essential  for  effective  operations. 

The  second  objective  relating  to  effective  team  training  really  emphasises  the  need  for 
effective  team  training  of  cognitive  rather  than  motor  skills.  The  DMT  system  purpose  is  to 
enable  defence  personnel  to  learn  and  practice  the  intellectual  skills  involved  in  complex 
air  operations  such  as  rapidly  analysing  incoming  data,  deducing  threat  intentions, 
coordinating  action,  and  optimising  system  employment.  DMT  will  not  provide  the 
required  motor  skill  training  needed  to  varying  degrees  by  all  aircrew  and  ground  crew, 
but  rather  has  the  primary  goal  of  team  learning  (Layton,  2000). 
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It  seems  that  DMT  has  potential  benefits  that  relate  specifically  to  crew  performance  and 
other  benefits  that  relate  to  operational/ organisational  performance.  Of  greatest  interest  to 
human  factors  researchers  are  the  benefits  that  relate  more  specifically  to  crew 
performance  as  improving  crew  performance  seems  to  be  a  logical  primary  goal  of  DMT, 
and  it  is  what  training  programs  aim  to  achieve  and  what  human  factors  researchers  can 
measure  and  evaluate  in  an  experimental  setting. 

There  are  a  number  of  potential  benefits  from  DMT  that  relate  to  crew  performance: 

1.  Firstly,  improved  opportunities  for  team  training  and  review  of  performance  and 
tactical  doctrine  -  Teamwork,  communication  and  coordination  training  can  be 
undertaken  in  a  manner  not  possible  with  live  exercises  as  events  can  be  frozen 
and  restarted  to  ensure  learning  points  are  understood.  DMT  can  be  used  for 
exploring  situations  that  have  occurred  in  exercises  for  further  evaluation  and 
education  adding  to  the  team  learning  value. 

2.  Crew  preparedness  -  Crews  should  be  better  prepared  for  the  unexpected  as  they 
would  have  experienced  a  wider  range  of  possible  combat  situations  than  is 
possible  during  normal  training  (Layton,  2000). 

3.  Maintenance  of  combat  proficiency  -  Potential  for  gaining  and  maintaining 
proficiency  levels  without  increasing  flying  hours. 

4.  Enhanced  reserve  aircrew  training  -  DMT  may  allow  the  maintaining  of  reserve 
crew  at  a  useful  currency  status  (Layton,  2000). 

Of  course  there  are  other  benefits  from  DMT  that  do  not  specifically  relate  to  crew 
performance  such  as: 

1 .  More  efficient  use  of  (limited)  actual  flying  hours  -  flying  hours  could  be  used  more 
for  refining  intellectual  skills  rather  than  for  their  initial  learning. 

2.  Trialing  of  new  equipment  -  the  research  and  development  that  DSTO  could 
undertake  to  support  air  operations  would  be  enhanced  by  being  able  to  analyse 
DMT  missions,  evaluate  proposed  tactical  changes,  and  trial  new  equipment  options 
(Layton,  2000). 

It  is  important  to  note  that  the  limited  documentation  available  concerning  DMT 
emphasises  issues  such  as  team  training,  intellectual  skills,  team  learning,  and  transfer  of 
training.  The  DSTO  approach  to  DMT  should  consider  that  the  training  system  to  be 
developed  should  ideally  foster  both  efficient  team  training  (i.e.,  team  learning  of 
cognitive  skills)  and  maintenance  and  attainment  of  combat  proficiency  and  readiness  via 
mission  rehearsal. 

Having  briefly  outlined  what  a  distributed  training  system  is  aiming  to  achieve,  the  report 
following  reviews  literature  that  is  relevant  to  the  attainment  of  these  goals.  Primarily,  this 
literature  concerns  issues  such  as  the  identification  of  key  competencies,  instructional 
strategies,  and  measures  of  performance  as  they  relate  to  team  functioning. 
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To  embark  on  a  review  of  the  training  and  team  performance  literature  is  indeed  a  sizeable 
task.  In  an  attempt  to  tie  together  the  areas  of  training  theory  and  practice,  it  is  worthwhile 
to  draw  upon  a  conceptual  framework  for  organising  the  training  research  proposed  by 
Cannon-Bowers  et  al.  (1991).  This  framework  aims  to  clarify  the  relationship  between 
various  areas  of  training  theory  and  between  theory  and  practice.  It  also  incorporates  three 
important  questions  relevant  to  training  research:  (1)  What  should  be  trained  (what  are  the 
knowledge,  skills,  and  attitudes  that  must  be  trained)?  (2)  How  should  training  be 
designed?  (3)  Is  training  effective,  and  why?  There  are  conceptual  and  theoretical 
developments,  as  well  as  training  techniques,  associated  with  each  of  these  questions.  In 
the  interests  of  clarity  and  providing  a  link  between  areas  of  research  and  practice,  the 
organisation  of  this  review  of  research  relevant  to  team  training  will  follow  the  framework 
described  by  Cannon-Bowers  et  al.  (1991). 
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2.  Competencies  and  cognitive  processes  for  team 

training 

It  is  important  to  reiterate  that,  at  this  point  in  time,  DMT  does  not  focus  on  manual 
control  skills,  and  as  such,  the  majority  of  this  section  focuses  on  higher-order  cognitive 
processes.  There  is  a  sizeable  body  of  literature  on  the  required  competencies  that  should 
form  the  basis  of  team  training.  These  competencies  will  be  the  initial  focus  of  this  section, 
but  then  discussions  will  move  to  higher-level  team  'processes'  and  associated  research. 

It  should  also  be  stated  that  virtually  none  of  the  team  training  literature  pertains  to 
distributed  teams,  that  is,  when  team  members  are  at  physically  different  locations.  While 
there  are  certainly  bound  to  be  some  differences  in  the  functioning  of  distributed  versus 
co-located  teams,  it  is  instructive  to  consider  what  is  already  known  about  how  co-located 
teams  function  before  considering  the  functioning  of  distributed  teams. 

2.1  Competencies  that  affect  teamwork 

Published  literature  over  the  last  decade  describes  three  major  components,  that  when 
combined,  define  teamwork.  These  competencies  are  cognitions  (knowledge-based 
competencies),  behaviour  (skill-based  competencies)  and  attitudes  (attitude-based 
competencies)  (Salas  &  Cannon-Bowers,  2000). 

It  is  worth  mentioning  here  some  of  the  basic  elements  that  make  up  a  team.  Indeed  there 
are  a  number  of  characteristics  that  have  been  established  in  the  literature  that  define  the 
characteristics  of  teams.  Some  of  these  are  that:  team  members  have  meaningful  task 
interdependencies;  team  members  hold  shared  and  valued  objectives/goals;  team 
members  are  hierarchically  organised;  team  members  hold  specialised  roles  and 
responsibilities,  team  members  use  multiple  information  sources;  team  members  possess 
adaptive  mechanisms;  and  team  members  perform  through  intensive  communication 
processes  (Salas  &  Cannon-Bowers,  2000). 

2.1.1  Cognitions 

Teamwork  has  a  cognitively-based  element  in  that  knowledge  is  required  in  order  to 
perform  the  team  tasks.  It  has  been  suggested  that  team  members  must  possess  compatible 
mental  models  (knowledge  structures)  of  their  team-mates'  roles,  the  tasks,  and  the 
situations  that  the  team  encounters,  in  order  to  be  effective  (Salas  &  Cannon-Bowers, 
2000).  These  knowledge  structures  create  expectations  and  allow  for  individuals  to  make 
predictions  concerning  how  to  perform  during  both  routine  and  novel  situations  (Cannon- 
Bowers,  Salas,  &  Converse,  1993). 

Research  also  suggests  that  team  members  should  possess  knowledge  about  the  purpose 
and  objectives  of  their  mission,  and  the  norms  that  are  expected  to  be  followed.  They  must 
also  understand  the  importance  of  teamwork,  their  individual  roles,  and  the 
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responsibilities  of  their  team-mates  (Cannon-Bowers  et  al.,  1993;  Cannon-Bowers, 
Tannenbaum,  Salas,  &  Volpe,  1995). 

Cue  strategy  associations  are  a  key  knowledge-based  strategy  (Cannon-Bowers  et  al.,  1993; 
Cannon-Bowers  et  al.,  1995).  That  is,  team  members  need  to  be  able  to  make  an  association 
between  a  set  of  cues  and  the  actions  that  need  to  be  undertaken  -  the  cues  in  the 
environment  should  trigger  team  members  to  behave  in  a  particular  way.  It  is  this  type  of 
knowledge  that  allows  team  members  to  perform  frequently  without  the  need  for  overt 
communication  (Salas  &  Cannon-Bowers,  2000). 

The  concepts  of  knowledge  structures,  mental  models,  and  other  cognitive  processes 
important  for  effective  team  functioning  are  pursued  in  detail  in  a  later  section  (section 
2.2).  The  aim  here  is  merely  to  provide  an  overview  of  the  different  types  of  competencies 
that  contribute  to  teamwork. 

2.1.2  Behaviour 

Team  members  require  the  skills  to  allow  them  to  perform  and  act  in  a  timely  and  accurate 
fashion.  As  such  there  has  been  considerable  research  directed  toward  uncovering 
consistent  behaviours  of  effective  teams.  There  are  actions  that  team  members  can  take 
that  have  been  shown  to  influence  teamwork.  For  example,  these  include:  performance 
monitoring;  performing  self-correction;  providing  tasks  and  motivational  reinforcement; 
using  closed-loop  communication;  adapting  to  unpredictable  situations;  being  assertive, 
and;  employing  implicit  or  explicit  coordination  of  activities.  According  to  Salas  & 
Cannon-Bowers  (2000)  the  skill-based  competencies  that  are  associated  with  effective 
teamwork  include  adaptability,  mutual  performance  monitoring,  team  leadership, 
communication,  assertiveness,  and  conflict  resolution. 

2.1.3  Attitudes 

Attitudes  about  the  task  and  other  team  members  will  have  a  significant  impact  on 
teamwork.  Willingness  to  remain  in  the  group,  team  morale,  beliefs  of  the  importance  of 
teamwork,  and  motivation  of  the  team  members  all  influence  the  effectiveness  of 
teamwork.  Collective  orientation,  being  the  disposition  to  receive  and  value  inputs  from 
others,  and  self-efficacy,  are  other  critical  factors  in  facilitating  teamwork. 

Teamwork  is  the  seamless  integration  of  related  knowledge,  skills,  and  attitudes  that  team 
members  use  to  optimise  performance.  It  is  important  to  have  an  understanding  of  these 
knowledge,  skills,  and  attitudes,  as  they  are  the  target  of  team  training,  which  of  course 
aims  to  improve  effective  teamwork  (Salas  &  Cannon-Bowers,  2000). 

Table  1  below  provides  some  generic  examples  of  cognitive,  behavioural,  and  attitudinal 
competencies  for  team  training. 
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2.1.4  Principles,  Guidelines,  and  Specifications 

Training  should  be  based  on  solid  principles  and  knowledge  of  human  learning,  and 
trainers  should  rely  on  theoretically  driven  and  empirically  validated  instructional 
principles  to  drive  the  instructional  design  process.  Salas  and  Cannon-Bowers  (2000) 
present  some  categories  of  principles  for  teamwork.  These  categories  are  general, 
adaptability-related,  communication-related,  performance  monitoring/ feedback-related, 
interpersonal  relations,  and  coordination/cooperation-related  principles.  More 
specifically,  those  authors  also  put  forward  a  number  of  general  principles  that  relate  only 
to  the  design  and  delivery  of  team-training  programs.  Salas  and  Cannon-Bowers  (2000) 
state  that  these  principles  are  supported  by  evidence  of  their  utility  and  effectiveness. 
Some  of  these  principles  of  team  training  are: 

•  Team  training  must  facilitate  information  presentation,  demonstration  of 
teamwork  behaviours,  practice,  and  feedback; 

•  Teams  must  be  provided  with  immediate  feedback  about  their  performance  during 
training; 

•  Team  members  who  require  transportable  competencies  (i.e.,  generic  competencies 
applicable  across  a  range  of  situations)  must  receive  training  at  the  individual 
level; 

•  Transfer  will  be  facilitated  when  individuals  receive  training  directed  at  individual 
skills  before  receiving  training  at  the  team  level; 

•  Transfer  will  be  facilitated  when  the  transition  from  individual  to  team  training  is 
based  on  the  formation  of  shared  knowledge  and  understanding  of  the  team  and 
its  tasks;  and 

•  Teams  that  need  to  develop  team-specific  competencies  from  either  team- 
contingent  or  context-driven  categories  must  receive  training  that  includes 
feedback  that  facilitates  the  formation  of  shared  or  common  expectations  for  task 
performance. 

Some  guidelines  for  team  training  are  also  put  forward.  These  guidelines  translate  the 
principles  into  practice,  and  thus  aid  in  the  design  and  development  of  team-training 
programs  (e.g.,  exercise  the  team  with  a  variety  of  novel  situations  to  build  an  adaptive 
repertoire).  These  guidelines  have  been  developed  to  provide  some  insight  into  the 
selection  of  appropriate  competencies  for  team  training  (i.e.,  not  all  competencies  will  be  of 
similar  importance  across  different  teams  and  tasks).  Some  competencies  may  be  specific 
to  a  particular  task  or  team,  or  generic  in  that  they  may  apply  across  teams  and  tasks.  In 
considering  the  competencies  that  need  to  be  trained,  it  is  important  to  consider  the 
stability  of  both  team  members  and  the  tasks  performed  over  time.  The  nature  of  team  and 
task  stability  will  align  the  competencies  required  to  one  of  the  four  categories  (Cannon- 
Bowers  et  al.,  1995).  Specifications  are  more  precise  statements  about  how  team  training 
should  be  designed,  and  state  specifically  what  to  do  and  when  to  do  it. 
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2.1.5  Summary 

The  preceding  four  sections  briefly  discuss  the  three  major  generic  types  of  competency 
thought  to  be  required  for  effective  team  functioning,  and  some  principles  and  guidelines 
to  promote  their  implementation  in  training  programs.  While  the  principles  and 
guidelines  are  intuitive  and  of  general  interest  value,  they  are  descriptive  only  and  do  not 
provide  any  solid  guidance  relating  to  how  to  develop  or  structure  a  team-training 
program.  Nonetheless  even  a  brief  examination  of  the  team  training  literature  yields  many 
references  discussing  team  competencies,  principles,  and  guidelines. 

While  there  have  been  many  competencies  identified  in  the  team  training  literature,  there 
are  some  that  appear  to  be  more  critical  than  others.  In  reviewing  the  team  work  literature, 
Dickinson  et  al.  (1992,  as  cited  in  Dickinson  &  McIntyre,  1997)  identified  and  defined7 
seven  core  components  of  teamwork  as  part  of  a  teamwork  model.  These  core  components 
are  as  follows: 

1)  Communication  is  a  major  component  of  teamwork  processes,  and  is  a  mechanism 
that  links  the  other  components  of  teamwork. 

2)  Team  orientation  refers  to  the  attitudes  that  team  members  have  toward  one 
another,  the  team  task,  and  the  team  leadership,  as  well  as  including  team 
cohesion. 

3)  Team  leadership  includes  the  structure  provided  by  the  designated  leaders  as  well 
as  other  team  members.  It  also  emphasises  the  importance  and  planning  to  support 
team  member  functions. 

4)  Monitoring  and  awareness  of  the  activities  and  performance  of  other  team 
members  is  also  an  important  team  process.  It  implies  that  individual  team 
members  have  a  good  understanding  of  the  tasks  of  other  team  members. 

5)  Feedback  is  critical  in  that  giving  and  receiving  of  feedback  among  team  members 
helps  teams  to  adapt  and  learn  from  their  performance. 

6)  Backup  behaviour  concerns  assisting  other  team  members  to  perform  their  tasks, 
which  assumes  a  degree  of  task  interchangeability  among  team  members,  and 
willingness  to  provide  and  seek  assistance. 

7)  Coordination  reflects  an  organised  execution  of  team  tasks  such  that  team 
members  respond  as  a  function  of  the  behaviour  of  others.  Coordinated  team 
performance  is  dependent  upon  effective  teamwork  dimensions  such  as 
communication,  monitoring,  and  backup. 

The  following  section  addresses  some  of  the  cognitive  team  processes  that  underpin  these 
dimensions  of  teamwork. 

2.2  Cognitive  Processes  for  Teams 

Teams  process  information,  make  decisions,  solve  problems,  and  make  plans.  Thorndike  & 
Klein  (1989,  as  cited  in  Klein  2000)  suggested  that  teams  can  be  considered  as  an  intelligent 
entity,  and  hence  it  is  possible  to  try  to  identify  the  key  cognitive  processes  required  by  the 
tasks  that  depend  on  teamwork.  The  cognitive  processes  important  for  teamwork  defined 
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and  discussed  by  Klein  (2000)  are:  control  of  attention;  shared  situation  awareness;  shared 
mental  models;  application  of  strategies  and  heuristics  to  make  decisions,  solve  problems, 
and  plan,  and;  metacognition.  These  processes  are  discussed  in  more  detail  in  the 
following  paragraphs. 

2.2.1  Control  of  attention 

The  control  of  attention  is  the  first  of  the  five  cognitive  processes.  In  a  team  context  it 
refers  to  the  way  that  information  is  managed.  Skilled  teams  learn  effective  information 
management  -  specifically  they  distribute  critical  information,  filter  irrelevant  messages, 
allocate  attention  to  important  functions,  and  seek  missing  information.  It  also 
encompasses  a  team's  working  memory,  being  the  way  it  uses  limited  resources  to  process 
inputs  in  parallel  (Klein,  2000).  Others  have  suggested  that  information  management 
should  be  regarded  as  being  one  requirement  of  resource  management,  which  also 
encompasses  management  of  cognitive  work,  communication,  and  action  (Orasanu,  1993). 
Nonetheless,  information  management  becomes  more  critical  as  teams  become  distributed 
(Klein,  2000). 

2.2.2  Shared  Situation  Awareness 

Team  situation  awareness  (SA)  has  been  described  as  being  one  of  two  types  of  team 
knowledge,  and  is  encompassed  in  the  team  situation  model  (Cooke  et  al.,  2000).  The 
second  form  of  team  knowledge,  encompassed  by  the  shared  mental  model,  is  discussed 
in  the  following  section. 

The  extent  to  which  team  members  have  the  same  interpretation  of  events  is  referred  to  as 
shared  SA.  SA  involves  interpreting  situation  cues  to  recognise  that  a  problem  exists 
which  may  require  a  decision  or  action — it  is  not  enough  to  simply  notice  the  presence  of 
cues  as  their  significance  must  also  be  appreciated  (Orasanu,  1993).  As  with  the  majority  of 
constructs,  shared  SA  is  more  important  in  dynamic  and  complex  environments  where 
there  is  greater  potential  for  team  members  to  develop  differing  interpretations  of  the 
situation  which  can  create  significant  problems.  Providing  all  team  members  with  all 
information  is  also  not  a  solution  as  this  will  undoubtedly  just  overload  the  team  members 
unnecessarily.  Thought  should  also  be  given  to  whether  it  might  actually  be  an  advantage 
for  individuals  to  have  slightly  different  interpretations  of  current  events.  That  is,  teams 
may  actually  benefit  from  team  members  having  differing  interpretations  of  the  situation. 
This  might  hold  in  situations  were  uncertainty  is  high,  so  even  if  the  team  leader  is 
mistaken  in  his/her  interpretation  there  would  be  a  greater  chance  of  other  team  members 
recognising  the  mistake,  and  depending  on  the  efficiency  of  information  management, 
rectifying  the  situation  accordingly  (Schraagen  et  al.,  2000). 

Before  discussing  team  SA  further,  it  is  important  to  consider  theoretical  perspectives  of 
individual  SA  so  as  to  provide  a  base  for  analysing  team  SA. 
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2.2.2.1  A  definition  and  model  of  situational  awareness 

There  has  been  much  research  and  discussion  about  the  concept  of  SA  in  the  past  decade. 
SA  is  founded  on  the  assumption  that  the  first  critical  step  for  successful  task  performance 
is  an  evaluation  of  the  'situation'  (Wickens,  Gordon,  &  Liu,  1998).  The  concept  of  SA  and 
its  importance  in  operator  performance  has  developed  to  such  a  point  that,  for  example,  in 
military  aviation,  SA  is  evaluated  on  nearly  every  training  event  because  it  is  seen  to  be 
critical  to  mission  success  (Oser,  2000). 

The  definition  of  SA  has  also  evolved  over  the  last  ten  years.  According  to  Endsley 
(1995b),  SA  is  "the  perception  of  the  elements  in  the  environment  within  a  volume  of  time 
and  space,  the  comprehension  of  their  meaning,  and  the  projection  of  their  status  in  the 
near  future"  (Endsley,  1995b).  SA  describes  a  variety  of  cognitive  processing  activities  that 
are  crucial  to  dynamic,  event-driven,  and  multitask  fields  of  practice.  Some  of  the 
cognitive  processes  that  may  be  invoked  under  the  label  of  SA  are  control  of  attention, 
mental  simulation,  directed  attention,  and  contingency  planning  (Sarter  &  Woods,  1995). 

According  to  Smith  and  Hancock  (1995),  SA  is  defined  as  adaptive,  externally  directed 
consciousness,  and  as  such  SA  generates  purposeful  behaviour  in  a  specific  task 
environment.  SA  represents  the  capacity  to  direct  consciousness  to  generate  competent 
performance  given  a  particular  situation  as  it  unfolds.  SA  is  seen  to  be  directly  related  to 
stress,  mental  workload,  and  other  energetic  facets  of  consciousness  (Smith  &  Hancock 
1995). 

Acquiring  and  maintaining  SA  becomes  increasingly  difficult  as  the  complexity  and 
dynamics  of  the  environment  increase.  In  dynamic  environments  many  decisions  are 
required  across  a  narrow  space  of  time,  and  performance  is  dependent  upon  an  ongoing 
and  up  to  date  analysis  of  the  environment.  The  state  of  the  environment  is  constantly 
changing,  often  in  complex  ways,  and  so  a  major  part  of  an  operators  role  is  obtaining  and 
maintaining  a  high  level  of  SA.  Operators  must  also  understand  what  they  are  perceiving, 
and  so  SA  incorporates  the  operators  understanding  of  the  situation  as  a  whole,  forming 
the  basis  for  decision  making  (Endsley,  1995b). 

In  terms  of  acquiring  and  maintaining  SA,  it  has  been  assumed  that  active  involvement  in 
a  task  would  promote  the  acquisition  of  SA.  Some  interesting  data  pertaining  to  SA  in  an 
analysis  of  311  incidents  in  commercial  aviation  were  published  recently  Oentsch,  Barnett, 
Bowers,  &  Salas,  1999).  This  study  found  that  the  captain  was  more  likely  to  lose  SA  when 
manipulating  the  controls  than  when  the  first  officer  was  the  pilot  flying.  Rather  than 
supporting  the  hypothesis  that  active  involvement  in  the  control  task  enhanced  SA,  the 
results  of  this  study  suggest  that  the  additional  workload  of  actually  flying  the  plane 
reduced  the  captain  s  ability  to  maintain  high  SA  of  the  global  scenario  in  a  problem 
situation.  That  is,  flying  the  aircraft  demands  cognitive  resources  and  thus  reduces  the 
resources  available  to  the  captain  for  building  and  maintaining  SA  (i.e.,  collecting, 
analysing,  and  communicating  information).  This  is  an  hypothesis  that  requires 
experimental  validation.  In  this  sense,  aviation  is  a  unique  field  in  that  the  team  leader  also 
regularly  engages  in  active  control  of  the  aircraft.  In  contrast,  the  captain  of  a  ship  would 
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rarely  engage  in  active  control  of  the  ship,  but  rather  would  provide  guidelines  for  junior 
team  members  to  execute  (Jentsch  et  al.,  1999). 

A  recent  model  of  SA  in  dynamic  systems  highlights  the  role  of  SA  in  system  performance 
(Endsley,  1995b).  This  model  clearly  shows  that  the  attainment  of  SA  occurs  prior  to  any 
decision  being  considered  or  made.  According  to  Endsley,  SA  is  comprised  of  three  levels. 
The  first  level  involves  perceiving  the  status,  attributes,  and  dynamics  of  the  key  elements 
in  the  environment.  For  example,  drivers  would  need  to  perceive  elements  such  as  other 
vehicles  or  obstacles  on  the  road,  and  their  dynamics,  relative  to  the  status  and  dynamics 
of  their  own  vehicle.  Level  2  SA  involves  a  comprehension  of  the  current  situation  that 
extends  beyond  being  aware  of  the  key  elements  that  are  present  in  the  environment  to 
gain  an  understanding  of  the  significance  of  those  elements  in  light  of  the  current  goals. 
This  model  suggests  that  synthesis  of  the  elements  from  level  1  allows  the  operator  to  form 
patterns  between  the  elements,  and  to  comprehend  the  significance  of  the  objects  and 
events  within  the  environment.  An  example  provided  by  Endsley  is  that  a  military  pilot 
must  comprehend  that  the  appearance  of  enemy  aircraft  within  a  certain  proximity  of  each 
other  and  in  a  particular  location  indicates  certain  things  about  their  objectives.  Operator 
experience  becomes  more  critical  for  level  2  SA.  An  inexperienced  operator  may  be  able  to 
identify  the  key  elements  in  the  environment  (level  1),  but  may  be  less  able  to  integrate 
those  elements  with  the  current  goals  to  form  an  adequate  comprehension  of  the  situation. 
The  third  and  highest  level  of  SA  is  the  projection  of  future  status.  The  ability  to  project  the 
future  action(s)  is  achieved  through  the  knowledge  of  the  status  and  dynamics  of  the 
elements  (level  1)  and  comprehension  of  the  situation  (level  2).  Referring  again  to  the  pilot 
example,  knowing  that  an  enemy  aircraft  is  in  a  particular  location  and  proximity  allows 
the  pilot  to  project  that  the  aircraft  is  a  threat  and  is  likely  to  attack  (level  2).  This 
awareness  provides  the  pilot  with  the  knowledge  necessary  to  decide  upon  the  most 
appropriate  course  of  action.  Analysis  of  Endsley's  model  shows  clearly  that  SA  goes  far 
beyond  merely  perceiving  information  in  the  environment.  SA  does  impact  upon  decision 
making,  and  also  affects  performance. 

There  are  a  number  of  factors  that  can  affect  SA,  which  can  be  classified  in  two  broad 
categories,  and  will  only  be  mentioned  briefly  here.  Human  properties  that  underlie  SA 
include  preattentive  processing,  attention,  perception,  working  memory,  long  term 
memory,  the  development  of  mental  models,  confidence,  automaticity  of  processing,  and 
goals.  There  are  also  task  and  system  factors  that  may  affect  SA,  which  include  system 
design,  interface  design,  stress  factors  (heat,  noise,  etc),  workload,  complexity,  and  level  of 
automation  (Endsley,  1995a). 

Managing  the  attentional  and  conceptual  processes  that  permit  strong  SA  involves 
significant  cognitive  resources,  and  as  such,  acquiring  and  maintaining  SA  must  be 
appreciated  as  an  integral  part  of  the  operators  mental  workload  (Adams,  Tenney,  &  Pew, 
1995). 
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2.2.2.1  Team  situation  awareness 

Team  SA  is  more  complex  than  combining  the  SA  of  individual  team  members  -  it 
involves  individual  SA  and  team  processes.  Models  of  both  individual  SA  and  team 
processes  are  dynamic,  meaning  that  the  relationship  between  the  two  is  not  simple.  To 
facilitate  the  development  of  team  SA,  team  members  need  to  have  information  that  will 
allow  them  to  develop  relevant  expectations  about  the  team  task  (Salas,  Prince,  Baker,  & 
Shrestha,  1995). 

Instructional  training  for  shared  SA  and  decision-making  is  not  well  developed  (Paris  et 
al.,  2000).  Each  team  member  perceives  one  or  more  parts  of  the  complex  environment, 
which  must  be  integrated  with  the  perceptions  of  other  team  members.  The  goal  is  to  have 
team  members  sharing  a  common  picture  of  the  environment,  with  the  possible  exception 
being  situations  of  high  uncertainty  (see  section  2.2.2).  Team  members  must  therefore 
communicate  effectively  to  promote  collective  awareness  of  the  environment  and  make 
timely  and  accurate  reports  of  any  potential  problems.  Training  that  develops  these 
abilities,  and  shared  perceptions  of  critical  task  dimensions,  will  improve  the  teams'  SA. 

While  there  is  much  about  team  SA  that  is  not  understood,  there  does  seem  to  be 
agreement  in  the  literature  that  at  least  one  dimension  of  teamwork,  communication,  is 
related  to  team  SA,  while  several  other  variables  such  as  planning,  self-critique,  and  task 
allocation  are  also  seen  as  important  (Salas  et  al.,  1995).  It  has  been  suggested  that  initially 
two  critical  measures  for  team  SA  should  relate  to  individual  SA  and  the  team  processes 
that  the  team  members  use  to  enhance  team  coordination.  Shared  mental  models 
(discussed  in  the  following  section)  are  thought  to  be  important  components  of  team  SA 
and  thus  compatibility  of  mental  models  among  team  members  should  be  measured 
(Stout,  Cannon-Bowers,  &  Salas,  1994).  For  the  team  training  of  SA  it  has  been  suggested 
that  training  should  focus  on  complex  communication  and  team  planning  (Salas  et  al 
1995). 

Perhaps  not  surprisingly  there  is  much  to  learn  about  what  and  how  to  train  for  team  SA. 
In  terms  of  what  to  train,  for  teams  with  high  turnover  it  has  been  suggested  that  training 
should  focus  on  task-specific  competencies,  that  is,  the  roles  in  the  team  and  the 
significance  of  those  roles  for  team  performance,  rather  than  on  competencies  for  a  specific 
team  (Cannon-Bowers  et  al.,  1995).  With  regard  to  how  to  train,  the  issues  are  how  to  train 
the  necessary  complex  communication  behaviours  needed  to  share  situation-relevant 
material,  and  how  to  develop  the  required  team  knowledge  (Salas  et  al.,  1995).  Practice 
with  feedback  has  been  identified  as  being  important  for  training  transfer,  although  the 
influence  of  length  of  practice,  type  of  tasks,  and  team  member  proficiency  on 
communication  behaviours  has  not  been  established  (Prince,  Chidester,  Bowers,  & 
Cannon-Bowers,  1992;  Salas,  Dickinson,  Converse,  &  Tannenbaum,  1992).  Finally,  further 
research  needs  to  resolve  the  issue  of  optimum  degree  of  overlap  in  team  member's 
perceptions  of  SA.  While  the  majority  of  the  literature  supports  the  belief  that  a  goal 
should  be  to  have  common  SA  amongst  team  members,  recent  work  has  suggested  that 
divergent  perceptions  of  SA  within  a  team  may  benefit  team  performance  in  particular 
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situations  such  as  those  involving  high  uncertainty.  Clearly  this  issue  needs  to  be  explored 
further  if  effective  training  programs  for  team  SA  are  to  be  developed. 

2.2.3  Shared  mental  model 

In  accordance  with  Klein  (2000)  the  third  cognitive  process  is  a  team's  shared  mental 
model.  It  should  be  stated  at  the  outset  that  the  terms  mental  model  and  knowledge 
structures  are  used  interchangeably  in  the  training  literature.  The  shared  mental  model 
represents  the  second  type  of  team  knowledge  as  described  by  Cooke  et  al  (2000). 

There  are  several  interpretations  of  the  concept  of  the  shared  mental  model.  Some 
researchers  define  mental  models  as  being  part  of  working  memory  and  constructed  from 
individual  background  knowledge,  while  others  suggest  that  mental  models  exist  within 
long-term  memory,  or  a  network  of  associations  between  domain  concepts  (Langan-Fox, 
Code,  &  Langfield-Smith,  2000). 

The  concept  of  the  mental  model  assumes  that  knowledge  is  organised  in  structured  and 
meaningful  patterns  that  are  stored  in  memory  (Johnson-Laird,  1983;  Rouse  &  Morris, 
1986).  Mental  models  allow  individuals  to  draw  inferences,  make  predictions,  and  to 
decide  upon  appropriate  actions  (Johnson-Laird,  1983).  There  have  been  several 
definitions  of  mental  models  offered  in  the  literature.  A  definition  that  appears  well 
accepted  is  that  mental  models  are  a  mechanism  that  enables  people  to  generate 
descriptions  of  system  purpose  and  form,  explanations  of  system  functioning,  and 
predictions  of  future  system  states  (Rouse  &  Morris,  1986).  Similarly,  Wickens  et  al.  (1998) 
suggest  that  mental  models  are  internal  representations  of  objects,  actions,  situations  or 
people,  built  on  experience  and  observation,  and  are  simulations  that  are  run  in  operators 
minds  that  allow  them  to  describe,  predict,  and  explain  behaviour. 

A  team  has  a  shared  mental  model  to  the  extent  that  team  members  have  the  same 
understanding  for  the  dynamics  of  key  processes.  These  processes  can  include  the  roles 
and  function  of  each  team  member,  the  nature  of  the  task,  the  use  of  equipment,  and  so  on. 
Of  particular  importance  is  the  degree  to  which  team  members  have  a  shared  mental 
model  of  their  own  roles  and  functions,  because  a  common  source  of  difficulty  occurs 
when  team  members  are  confused  about  who  is  doing  what.  The  shared  mental  model 
refers  to  the  configuration  of  the  team  and  the  way  it  is  supposed  to  perform  routines, 
whereas  shared  SA  refers  to  the  current  and  projected  situation  (Cannon-Bowers,  Salas,  & 
Converse,  1993;  Schraagen,  Chipman,  &  Salin,  2000). 

A  team  mental  model  reflects  the  dependencies  or  interrelationships  between  team 
objectives,  team  mechanisms,  temporal  patterns  of  activity,  individual  roles  and  functions, 
and  relationships  between  individuals.  Shared  mental  models  therefore  allow  crew 
members  to  implicitly  and  more  effectively  coordinate  their  behaviours,  as  they  are  better 
able  to  recognise  the  individual  responsibilities  and  information  needs  of  fellow  crew 
members,  monitor  their  activities,  diagnose  deficiencies,  and  provide  support,  guidance, 
and  information  as  needed  (Paris  et  al.,  2000). 
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Research  is  not  conclusive  regarding  what  should  be  the  content  of  shared  mental  models, 
and  the  nature  and  extent  of  information  that  must  be  shared  to  allow  the  team  to  form 
accurate  explanations  and  expectations  of  team  performance.  At  what  point  do  team 
member's  knowledge  and  expectations  overlap  so  much  that  the  uniqueness  of  individual 
contributions  is  lost?  (Cannon-Bowers  et  al.,  1993). 

Human  factors  and  ergonomics  specialists  have  developed  a  heavy  reliance  upon  mental 
models.  The  belief  is  that  if  the  mental  models  from  operators  in  particular  situations  can 
be  communicated  to  the  system  designers,  then  the  systems  can  be  designed  with 
interfaces  that  better  match  the  needs  and  expectations  of  the  operators  (Wilson,  2000). 
Similarly,  training  systems  can  be  designed  to  build  up  mental  models  consistent  with  that 
of  the  expert.  As  discussed  later,  shared  mental  models  also  form  the  basis  for  theoretical 
frameworks  of  team  decision-making  (Orasanu  &  Salas,  1993). 

2.2.4  Strategies 

While  not  a  process  per  se,  Klein  (2000)  states  that  a  team  develops  strategies  and  heuristics 
to  make  decisions  and  problem  solve  as  it  gains  experience.  Despite  lists  of  steps  and 
procedures,  most  teams  will  learn  strategies  (and  shortcuts)  that  are  not  in  the  procedures. 
That  is,  a  skilful  team  will  learn  who  the  key  decision  makers  and  sources  of  expertise  are. 
Klein  (2000)  suggests  that  if  we  are  to  attempt  to  better  understand  team  functioning  then 
is  it  important  to  consider  the  strategies  that  the  team  has  developed  for  moving 
information  and  ideas,  and  for  assimilating  the  various  sources  of  information. 

2.2.5  Metacognition 

The  final  process  is  metacognition,  which  encompasses  a  teams  need  to  monitor  itself  and 
determine  when  difficulties  are  encountered,  how  the  difficulty  is  linked  to  its 
vulnerabilities  an<^  *irnitations' anc*  w^ere  it  needs  to  shift  its  strategies  (Schraagen  et  al.. 

Metacognition  can  refer  broadly  to  a  reflective  executive  function,  which  encompasses 
defining  a  problem,  devising  a  plan  to  solve  it,  and  deciding  what  information  and 
resources  are  both  required  and  available  (Orasanu,  1993).  Effective  teams  can  self¬ 
monitor  and  make  changes.  Ineffective  teams  usually  are  unaware  why  they  are  getting  in 
their  own  way  and  so  are  in  no  position  to  adapt  operations  to  improve  performance 
(Schraagen  et  al.,  2000).  It  w ould  appear  intuitive  that  the  process  of  metacognition  would 
be  more  effective  when  the  team  had  well  developed  shared  mental  models  and  SA. 

2.2.6  Team  Decision-Making 

Decision-making  is  another  key  team  process,  and  the  literature  concerning  team  decision¬ 
making  invokes  many  of  the  team  processes  discussed  above,  such  as  shared  SA,  shared 
mental  model,  and  metacognition  (Orasanu,  1993),  and  so  is  worth  talking  about  here.  This 
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highlights  one  of  the  difficulties  in  assimilating  the  literature  on  these  cognitive  processes 
in  that  there  is  a  considerable  degree  of  overlap  between  them. 

Experts  come  to  see  the  world  in  terms  of  patterns  that  develop  with  experience.  This 
reduces  workload  but  also  facilitates  more  rapid  action.  Given  that  pattern  recognition  is 
important,  how  should  it  be  trained?  Simulation  can  be  used  because  it  can  create  many 
practice  trials  that  are  designed  to  build  up  the  recognition  of  patterns  that  can  be 
constantly  adapted  to  the  trainee  in  terms  of  the  level  of  difficulty  for  instructional 
purposes  (Means,  Salas,  Crandall,  &  Jacobs,  1993).  Means  et  al.  (1993)  suggest  that  training 
of  metacognitive  skills  (reflection  upon,  and  regulation  of,  one's  own  thinking)  may  be  the 
best  option  for  generalisable  skills  that  will  aid  team  decision-making. 

The  concept  of  shared  mental  models  becomes  a  means  of  defining  the  determinant  of 
team  attentional  capacity.  If  shared  models  are  consistent  then  team  capacity  should  be 
enhanced  because  there  will  be  less  process  loss.  If  they  are  inconsistent  then 
misunderstandings  and  thus  failures  may  occur  (Duffy,  1993). 

2.2.6.1  Naturalistic  Decision-Making 

In  the  previous  decade  there  has  been  a  paradigm  shift  in  decision-making  research,  with 
a  shift  towards  the  study  of  decision-making  in  naturalistic  settings  (NDM).  This  shift  is 
away  from  classical  decision-making  research  which  was  seen  as  often  focussing  on  sterile 
and  contrived  decision-making  situations  with  results  that  were  of  little  consequence  to 
real  world  decision  makers  (Beach  &  Lipshitz,  1993).  The  focus  has  shifted  to  study 
decisions  that  are  embedded  within  complex  tasks  and  that  are  made  by  competent  and 
experienced  decision  makers. 

A  number  of  factors  have  been  identified  that  characterise  decision  making  in  naturalistic 
environments  (Orasanu,  1993).  Some  or  all  of  these  factors  need  to  be  present  for  a 
decision  to  be  considered  naturalistic.  These  factors  are:  ill-structured  problems;  uncertain, 
dynamic  environments;  shifting,  ill-defined,  or  competing  goals;  multiple  event-feedback 
loops;  time  constraints;  high  stakes;  multiple  players;  and  organisational  norms  and  goals 
that  must  be  balanced  against  the  decision  makers'  personal  choice. 

The  NDM  approach  has  made  some  important  contributions  to  the  study  of  decision 
making  (Klein,  1993).  It  has  emphasised  how  decision  makers  bring  their  experience  to  the 
fore  in  making  a  decision,  it  broadens  the  focus  of  decision-making  research,  and  the 
models  associated  with  NDM  emphasise  the  different  cognitive  strategies  and  processes 
that  are  used  when  a  decision  situation  is  viewed  as  a  temporal  (rather  than  static)  event. 
Moreover,  NDM  has  provided  researchers  with  a  new  method  for  studying  decision¬ 
making  processes. 

A  major  theory  in  the  area  is  the  recognition-primed  decision  model  (RPD)  which 
describes  how  experienced  operators  make  decisions  in  natural  settings  (Klein,  1993).  The 
decision  maker  identifies  the  situation  as  typical,  which  allows  the  person  to  know  certain 
points,  such  as  which  goals  make  sense,  which  cues  are  relevant,  what  to  expect,  and 
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which  actions  typically  work.  The  decision  maker  can  then  implement  an  appropriate 
course  of  action  without  further  deliberation.  For  more  complex  situations,  the  decision 
maker  can  appraise  the  possible  courses  of  action  by  choosing  the  first  one  and  running 
mental  simulations  and  evaluating  how  that  action  will  unfold  in  the  given  context.  In 
most  cases  it  is  the  first  chosen  course  of  action  that  is  simulated  and  accepted  to  be  put 
into  action.  Support  for  Klein  s  RPD  model  is  found  in  studies  examining  decision  making 
of  pilots  (Endsley  &  Smith,  1996)  and  offshore  oil  workers  (Flin,  Slaven,  &  Stewart,  1996)  to 
name  a  few.  Klein's  RPD  model  emphasises  the  importance  of  SA  for  successful  decision 
making  in  field  settings. 

This  model  does  have  applied  implications,  particularly  for  training  interventions  to 
facilitate  the  transition  from  novice  to  expert  decision  maker.  Since  pattern  recognition  is  a 
key  element  in  the  decision  making  process  (Klein,  1993),  it  has  been  suggested  that 
interface  training  interventions  support  the  development  of  highly  effective  and  robust 
heuristic  strategies  rather  than  the  development  of  abstract  and  cognitively  intensive 
strategies  (Kirlik,  Walker,  Fisk,  &  Nagel,  1996).  These  authors  were  able  to  show  the 
benefits  to  decision  making  skill  acquisition  from  training  techniques  that  involved 
concrete  situational  patterns  in  the  task  environment,  and  from  augmented  displays  that 
enhance  critical  perceptual  properties  and  relationships  during  training. 

The  importance  of  metacogmtion  in  decision-making  is  emphasised  in  the 
recognition /metacognition  (RM)  model  of  decision  making  (Cohen,  Freeman,  &  Wolf, 
1996).  This  model  appears  to  be  an  extension  (of  sorts)  from  Klein's  model  of  decision 
making  which  was  based  largely  on  recognitional  processes  (Klein,  1993).  This  model 
highlights  two  strategies:  recognition  of  expectations  and  associated  responses;  and  the 
critiquing  and  correcting  of  those  responses.  This  latter  point  is  linked  with  the  concept  of 
metacognition,  that  is,  processes  that  monitor  and  regulate  other  thought  processes  (such 
as  memory  and  attention).  Metacogrutive  skills  also  include  verifying  and  improving  the 
results  of  pattern  recognition  as  part  of  the  decision  making  process  in  uncertain 
situations.  According  to  the  RM  model,  as  decision  makers  become  more  familiar  with  a 
domain,  they  acquire  abstract  knowledge  about  the  types  of  events  and  relationships 
between  events  that  are  relevant  in  particular  situations.  As  such,  when  confronted  with  a 
new  situation,  decision  makers  can  draw  upon  this  generic  knowledge  to  integrate  the 
new  information  and  subject  the  results  to  repeated  modification  and  evaluation. 

With  regard  to  team  decision-making,  the  NDM  perspective  has  helped  by  focussing 
attention  on  the  processes  involved  in  team  decision-making.  Other  principles  that  have 
arisen  from  NDM  that  can  be  used  to  develop  an  expert  team  are:  fostering  shared  mental 
models  of  task  work  and  team  member  roles;  training  skills  such  as  SA,  communication, 
leadership,  and  adaptability;  providing  guided  practice  in  naturalistic  conditions;  linking 
cue  patterns  with  response  strategies  with  regard  to  different  tasks  and  team  members; 
and  training  team  leaders  to  maintain  shared  SA  by  providing  periodic  updates  to  team 
members  (Salas,  Cannon-Bowers,  &  Johnston,  1997). 
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NDM  theories  imply  that  context-specific  domain  knowledge  is  a  crucial  aspect  of  expert 
decision-making,  and  as  such,  training  decision-makers  in  problem  solving  specific  to  a 
given  domain  may  be  more  fruitful  than  training  generic  problem  solving  skills  (Cannon- 
Bowers  &  Salas,  1997b). 

2.2.7  Team  processes  for  distributed  teams 

The  majority  of  team  research  has  involved  teams  that  are  located  in  the  same  physical 
space.  DMT  is  likely  to  involve  both  co-located  and  distributed  teams.  The  distributed 
nature  of  teams  introduces  a  further  level  of  abstraction  that  team  members  must  deal 
with,  which  has  been  referred  to  as  'team  opacity'  (Fiore,  Salas,  &  Bowers,  2001).  In  the 
context  of  distributed  teams  Fiore  et  al  (2001)  use  this  term  to  describe  the  decreased 
awareness  of  team  member  actions  that  may  alter  their  interactions.  In  co-located  teams 
the  interactions  between  individuals  are  transparent  in  that  is  it  easier  to  assess  the  actions 
of  other  team  members,  but  the  demands  on  working  memory  in  non  co-located  teams  are 
higher.  At  the  team  level,  physical  separation  introduces  complexity  because  a  team 
process  such  as  communication  is  reliant  upon  implicit  and  explicit  cues,  however  there  is 
likely  to  be  a  much  heavier  reliance  on  explicit  cues  in  distributed  teams  (whereby 
gestures  may  be  sufficient  in  co-located  teams  for  some  forms  of  distributed  team 
communication)  (Fiore  et  al.,  2001).  The  interactions  between  co-located  and  distributed 
teams  are  likely  to  be  quite  different. 

While  there  is  some  understanding  of  team  processes  such  as  communication  and  shared 
mental  models,  considerably  less  is  known  about  distributed  cognition,  that  is,  what 
information  a  distributed  team  uses  to  effectively  perform  the  task,  and  how  the 
placement  of  team  members  in  separate  locations  may  alter  the  requirements  of  effective 
(co-located)  teams.  Fiore  et  al.  (2001)  suggest  that  perhaps  the  development  of  shared 
mental  models  in  distributed  teams  is  the  most  significant  consequence  of  the  distributed 
interactions,  as  the  development  shared  mental  models  is  heavily  reliant  upon 
communication. 

Fiore  et  al.  (2001)  therefore  suggest  that  increased  emphasis  should  be  placed  on 
communications  that  occur  outside  of  the  distributed  exercise,  that  is,  pre  and  post¬ 
exercise.  They  suggest  that  distributed  interactions  may  be  facilitated  by  pre-exercise 
coordination  efforts,  and  group  development  exercises  and  cross-training  may  be 
appropriate  here  (cross-training  is  discussed  in  more  detail  in  section  3.2.2).  Post-exercise 
coordination  efforts,  such  as  team  self-correction,  may  also  be  helpful  in  the  development 
of  team  processes,  but  may  also  help  the  development  other  important  team  attitudes  such 
as  trust  and  cohesion  (Fiore  et  al.,  2001).  The  lack  of  literature  in  the  field  indicates  that 
research  examining  the  factors  that  contribute  to  the  effective  functioning  of  distributed 
teams  is  in  its  infancy. 

Critical  processes  for  team  functioning  have  been  discussed,  and  well  as  the  difficulties 
facing  distributed  teams.  The  question  now  is  how  to  go  about  identifying  and  defining 
these  cognitive  processes  for  complex  performance. 
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2.2.8  The  analysis  of  cognitive  processes 

The  analysis  of  complex  skills  and  their  associated  knowledge  requirements  is  a  major 
problem  for  training.  Taxonomies,  such  as  Hierarchical  Task  Analysis  (HTA),  are  useful, 
but  use  relatively  global  categories  and  do  not  include  how  to  analyse  the  detailed 
psychological  components  of  skill  (Patrick,  1991). 

There  are  many  types  of  knowledge  that  underpin  performance  of  a  complex  task.  All  of 
these  types  of  knowledge  need  to  be  identified,  described,  and  assembled  in  order  to 
produce  a  complete  analysis.  There  are  many  constructs  in  cognitive  psychology  to 
describe  knowledge  representation.  It  is  well  beyond  the  scope  of  this  report  to  review 
these  cognitive  concepts  and  so  they  will  be  mentioned  only  briefly  here.  Further  details 
can  be  found  in  texts  on  cognitive  psychology. 

In  procedural  knowledge  the  concept  of  a  schema  (a  cognitive  mechanism  that  helps  in  the 
organisation  of  knowledge)  is  important  in  explaining  how  experiences  are  organised  in 
memory,  how  new  information  is  assimilated,  and  how  actions  are  organised  in  familiar 
situations.  The  acquisition  of  rules  has  been  another  means  by  which  to  characterise 
cognitive  performance  (e.g.,  Anderson,  1982).  More  recently  the  concept  of  the  mental 
model  has  been  invoked  to  encompass  anything  from  the  simulation  of  a  cognitive  activity 
to  something  resembling  the  notion  of  a  schema  (Johnson-Laird,  1983;  Rouse  &  Morris, 
1986).  Other  approaches  have  described  rules  of  thumb  or  heuristics  that  experts  use  in 
performing  a  complex  cognitive  task  which  need  to  be  verbalised  to  facilitate  the 
progression  from  novice  to  expert  performer.  Clearly  there  are  many  types  of  knowledge 
involved  in  the  performance  of  complex  cognitive  tasks  and  many  concepts  invoked  in 
cognitive  psychology  to  account  for  them.  There  is  however  no  clear  mapping  between  the 
type  of  task  to  be  analysed  and  the  psychological  constructs  used  (Patrick,  1992). 

The  analysis  of  a  cognitive  task  should  result  in  a  model  of  how  an  expert  performs  that 
task,  and  the  different  types  of  knowledge  involved  and  their  associated  representations. 
This  is  difficult  because  experts  have  the  ability  to  switch  between  different  types  of 
knowledge  and  use  them  almost  simultaneously  during  performance  of  a  task.  Rasmussen 
(1986)  has  made  some  progress  in  this  area  in  considering  the  human  operator  in  complex 
systems.  He  identified  two  dimensions  that  can  be  used  to  map  types  and  levels  of 
knowledge  involved  in  performance.  The  first  dimension  of  aggregation  refers  to  the  size 
of  the  unit  under  consideration,  for  example,  whole  system  through  to  individual 
components.  The  second  dimension  is  abstraction,  of  which  there  are  five  levels:  system 
purpose  relates  to  objectives  and  constraints  of  the  system;  abstract  function  represents  the 
function  of  the  system;  generalised  function  includes  description  in  terms  of  standard 
functions  and  process;  physical  function  includes  mechanical,  electrical,  etc  processes  of 
the  system  and  part;  and  physical  form  is  the  actual  physical  appearance  and 
configuration  of  the  system  and  parts.  Rasmussen  (1986)  provides  examples  of 
descriptions  in  the  means-ends  abstraction  hierarchy  for  manufacturing  and  computer 
systems.  The  operator  moves  between  different  levels  during  task  performance,  as 
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determined  by  the  nature  of  the  task,  and  different  types  of  knowledge  are  required  to 
support  performance  at  each  level.  The  dimensions  of  aggregation  and  means-ends  can  be 
combined  to  show  the  progression  of  cognitive  behaviour  during  the  performance  of  a 
task  (see  Patrick,  1992). 

It  is  however  not  possible  to  perform  a  full  and  coherent  psychological  analysis  of  a 
complex  task.  To  understand  human  expertise  more  generally  will  involve  more  empirical 
and  simulation  research  (Anderson,  1982).  In  light  of  this  it  would  seem  that  a  mixture  of 
top-down  and  bottom-up  approaches  might  be  optimal  at  this  point. 

2.2.8.1  Approaches  to  Cognitive  Task  Analysis 

The  grouping  of  required  competencies  in  section  2.1  was  cognitive,  behavioural,  and 
attitudinal.  Of  great  interest  in  this  review  are  the  cognitive  competencies.  Behavioural 
accounts  of  team  performance  identify  which  team  member  is  responsible  for  each 
subtask,  show  how  responsibility  for  a  subtask  changes  through  the  performance  cycle, 
and  helps  in  the  observation  of  the  team  and  whether  the  team  is  following  procedures. 
Behavioural  accounts  do  not  however  help  in  understanding  how  the  team  was 
interpreting  the  situation,  how  the  team  made  decisions,  how  team  members  might  have 
been  confused  about  their  roles  and  functions  and  those  of  others,  or  how  the  team  self¬ 
monitors  so  it  could  adapt  when  necessary  (Schraagen  et  al.,  2000).  Cognitive  approaches, 
such  as  Cognitive  Task  Analysis  (CTA),  can  help  to  describe  the  way  a  team  is  thinking 
rather  than  the  steps  it  is  following.  This  information  can  then  be  used  to  inform  the 
development  of  training  programs. 

There  are  many  different  CTA  techniques  that  have  been  used  to  identify  the  cognitive 
demands  and  activities  underlying  task  performance  (e.g.,  Schraagen  et  al,  2000;  Kirwan 
&  Ainsworth,  1992;  Gordon  &  Gill,  1997).  Some  of  these  techniques  involve  unstructured 
and  structured  interviews  with  subject  matter  experts  (e.g.,  Militello  &  Hutton,  1998), 
knowledge  elicitation  procedures  (Fowlkes,  Salas,  Baker,  Cannon-Bowers,  &  Stout,  2000; 
Hoffman,  Crandall,  &  Shadbolt,  1998),  whilst  others  involve  more  formal  modelling 
procedures.  The  diversity  in  CTA  approaches  potentially  leads  to  confusion  about  the  type 
of  results  that  are  expected  from  CTA  and  how  the  results  will  impact  upon  system  design 
or  evaluation.  An  important  issue  is  that  approaches  to  CTA  are  weakly  linked  to  the 
design  and  development  of  advanced  decision  support  systems  (Potter,  Roth,  Woods,  & 
Elm,  2000). 

One  form  of  CTA  is  the  critical  decision  method  (CDM),  which  attempts  to  elicit  expert 
knowledge  via  retrospection.  As  such,  to  retrospect  there  must  be  an  event  about  which 
one  can  retrospect.  In  distributed  environments,  often  a  particular  operator  may  not  have 
accurate  knowledge  about  their  performance,  i.e.,  information  from  their  element  of  the 
task  may  be  passed  onto  another  team  member,  but  the  operator  may  not  know  if  the 
information  they  provided  was  correct.  As  such  it  is  possible  that  the  operator  maybe  less 
able  to  refine  their  performance  and  develop  expertise.  People  can  have  difficulty  recalling 
incidents  when  feedback  is  lacking,  which  is  not  favourable  for  the  CDM  method 
(Hoffman  et  al.,  1998). 
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Recently  it  has  been  suggested  that  CTA  is  more  than  the  application  of  any  single 
technique,  and  that  developing  a  more  meaningful  understanding  of  a  domain  relies  on 
multiple  converging  techniques  (Potter  et  al.,  2000).  They  contend  that  the  focus  of  CTA  is 
on  building  a  model  that  captures  the  analysts'  understanding  of  the  task  domain,  the 
domain  practitioners'  knowledge  and  strategies,  and  how  existing  artefacts  influence 
performance.  More  specifically.  Potter  et  al.  (2000)  refer  to  CTA  as  a  bootstrapping  process 
that  is  focussed  on  understanding  both  the  domain  (mapping  the  cognitive  demands  of 
the  fields  of  practice)  and  practitioners  (modelling  expertise  and  cognitive  strategies) 
through  a  series  of  complementary  (empirical  and  analytical)  methods. 

It  is  important  to  understand  the  domain  characteristics  and  the  cognitive  demands  they 
impose  because  it  provides  a  framework  for  interpreting  practitioner  performance 
(what  complexities  in  the  domain  are  they  responding  to?  What  constraints  of  the  domain 
are  they  less  sensitive  to?),  and  it  helps  define  requirements  for  effective  support  (what 
aspects  of  performance  could  use  support?)  and  clarifies  the  bounds  of  feasible  support 
(what  technologies  can  be  used  to  support  the  complexities  of  the  domain?).  The  second 
major  issue  to  understand  is  how  the  practitioner  responds  to  the  demands  of  the  domain. 
To  understand  effective  strategies  for  managing  demands  of  a  domain  it  is  critical  to 
understand  the  knowledge  and  strategies  that  expert  practitioners  have  developed  in 
response  to  those  demands.  This  understanding  can  highlight  where  support  is  needed 
(Potter  et  al.,  2000). 

An  integrated  approach  to  CTA  generally  requires  the  use  of  multiple  converging 
techniques  including  those  that  focus  on  understanding  the  demands  of  the  domain  as 
well  as  those  that  focus  on  understanding  the  knowledge  and  strategies  of  domain 
practitioners  (Potter  et  al.,  2000).  These  are  different  approaches  that  provide 
complementary  perspectives.  Potter  et  al.  (2000)  believe  that,  if  resources  are  scarce,  that  it 
is  more  effective  to  use  techniques  that  sample  both  perspectives,  even  if  cursorily,  than  to 
use  all  resources  for  one  technique.  The  use  of  multiple  techniques  is  more  likely  to 
uncover  complexities  and  surprises,  and  converging  results  increase  the  confidence  of 
understanding. 

In  order  to  overcome  the  problems  associated  with  conducting  CTA  with  new 
technologies  (i.e.,  in  domains  that  do  not  exist).  Potter  et  al.  (2000)  advocate  that  the  CTA 
process  can  be  extended  into  the  design  and  prototype  development  phase.  An  example  of 
how  this  process  was  implemented  is  provided  by  Potter  et  al.  (2000)  and  will  only  be 
mentioned  briefly  here  for  illustrative  purposes.  The  example  describes  a  CTA  to  provide 
design  concepts  for  a  graphic  user  interface  for  a  next  generation  decision  support  system. 

From  a  training  perspective,  specification  of  performance  goals  enables  the  identification 
of  the  knowledge  required  to  achieve  those  goals.  Goals  can  be  used  as  the  framework  for 
a  top-down  analysis,  and  can  be  divided  into  subgoals  to  unravel  more  specific  strategies 
and  knowledge  requirements.  The  best  known  descriptive  model  that  adopts  a  top-down 
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goal-oriented  approach  is  that  proposed  by  Rasmussen  (1986)  for  die  analysis  of  decision¬ 
making  tasks  (Patrick,  1991). 

The  first  step  of  the  CTA  in  this  case  study  was  to  build  a  goals-means  representation  of 
the  domain  (as  per  Rasmussen,  1986;  Vicente,  1999).  This  functional  representation 
specifies  the  underlying,  unchanging  system  functions  and  relationships  that  form  the 
basis  for  system  complexity  and  goal  achievement.  The  domain  model  is  critical  for 
understanding  the  context  (goals  to  be  achieved,  strategies  for  achieving  the  goals)  in 
which  the  practitioner  must  perform.  It  also  is  intended  to  provide  a  robust  model  of  a 
complex  environment,  independent  of  specific  events,  tasks,  and  strategies.  A  claimed 
major  benefit  is  that  it  enables  identification  of  person-machine  and  inter-person 
information  requirements  to  support  decision  making  and  problem  solving  in 
unanticipated  situations  that  system  designers  have  not  explicitly  foreseen  or  addressed.  It 
provides  a  starting  point  in  understanding  fundamental  relationships,  scope  of  the 
application,  and  system  objectives.  The  second  step  involved  interviews  with  potential 
users  of  the  system  to  understand  their  information  needs,  knowledge,  and  strategies.  The 
primary  use  of  the  functional  representation  in  this  second  step  was  as  a  mechanism  for 
resolving  discrepancies  between  different  interviews.  Step  3  involved  refining  the  initial 
functional  representation  in  light  of  the  interviews  conducted  in  step  2.  The  functional 
model  therefore  begins  to  capture  some  of  the  practitioner  expertise,  knowledge,  and 
strategies.  In  step  4,  the  interviews  are  also  used  to  identify  critical  decisions  within  the 
work  domain,  which  must  then  be  supported  in  the  resulting  system  design.  Supporting 
information  needs  and  visualisation  design  (to  support  the  information  needs  for  each  of 
the  critical  decisions)  are  the  final  two  steps.  While  only  mentioned  briefly  here,  this 
example  highlights  how  multiple  converging  techniques  can  be  used  in  conducting  CTA. 

While  the  abovementioned  techniques  have  been  well  documented  for  individual  tasks,  on 
the  whole  the  approaches  have  not  been  well  developed  for  a  cognitive  task  analysis  for 
team  tasks.  The  ability  to  identify  the  specific  behaviours  that  comprise  teamwork  is  a 
prerequisite  for  effective  team  training,  and  several  researchers  have  documented 
concerns  regarding  the  validity  of  using  measurement  instruments  derived  from 
individuals  in  team  performance  research  (Bowers,  Baker,  &  Salas,  1994;  Dyer,  1984). 
Hence  a  clear  understanding  of  the  team  behaviours  and  processes  is  required. 

Few  techniques  have  been  used  to  identify  team  skills  and  processes.  Annett, 
Cunningham,  &  Mathias-Jones  (2000)  describe  the  use  of  Hierarchical  Task  Analysis  for 
Teams  (HTAT)  to  identify  team  skills  required  by  anti-submarine  warfare  teams.  The 
HTAT  approach  involves  decomposing  team  goals  into  subgoals,  and  then  identifying 
subgoals  which  can  only  or  best  be  achieved  by  teamwork  (Annett  &  Cunningham,  2000; 
Shepherd,  1998).  Teamwork  was  defined  as  communication  between,  and  the  coordination 
of  actions  of,  members  of  the  team. 

Perhaps  the  best  technique  to  capture  the  five  cognitive  processes  for  teams  discussed 
earlier  and  defined  by  Klein  (2000)  is  a  team  CTA.  Klein  and  colleagues  have  used  this 
technique  to  examine  the  cognitive  processes  in  US  Marine  Corps  command  posts  and 
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nuclear  power  plant  emergency  response  teams.  Typically  a  team  CTA  is  conducted 
primarily  via  interviews  with  subject  matter  experts,  however  both  observations  and 
simulations  of  team  behaviour  can  also  be  used.  The  goal  of  team  CTA  is  to  collect 
information  concerning  the  manner  in  which  teams  make  decisions  and  the  way  the  team 
is  managed.  By  gaining  an  understanding  of  these  issues  it  is  possible  to  further  explore 
the  five  team  processes  in  more  detail.  Specifically,  a  team  CTA  needs  to  address  how  the 
team  manages  information,  how  it  attempts  to  maintain  shared  SA,  the  extent  to  which 
mental  models  are  shared,  the  strategies  and  expertise  the  team  has  developed  and 
learned,  and  the  extent  to  which  the  team  is  able  to  identify  and  manage  problems. 

Finally,  Cognitive  Work  Analysis  (CWA)  is  another  form  of  CTA  that  is  used  to  model, 
analyse,  and  design  complex  systems  (Rasmussen,  Pejtersen,  &  Goodstein,  1994;  Vicente, 
1999).  The  first  stage  of  CWA  is  Work  Domain  Analysis  (WDA).  WDA  is  conducted  at  the 
functional  (rather  than  behavioural)  level  and  describes  the  boundaries  of  the  work 
domain  that  shape  behaviour.  Within  the  boundaries  of  the  work  domain  the  operator 
uses  the  available  resources  to  engage  in  a  variety  of  actions  in  order  to  achieve  the  system 
goals.  By  focussing  on  the  functional  structure  of  a  system,  WDA  is  independent  of  task- 
type  or  specific  behaviours,  and  is  therefore  valid  across  a  range  of  situations  (Vicente, 
1999).  As  such  WDA  is  suitable  for  military  environments  in  which  a  number  of  different 
tasks  may  be  performed  in  order  to  reach  a  particular  goal,  and  where  initiative  and 
adaptive  behaviour  are  encouraged  (Braithwaite,  2002). 

WDA  identifies  the  functional  structure  of  a  system,  and  thus  provides  a  framework  for 
designing  training  systems  that  support  adaptive  behaviour.  If  a  training  system  is 
modelled  on  the  functional  structure  of  the  work  domain,  then  the  training  system  would 
be  able  to  provide  trainees  with  the  same  possibilities  for  action  that  would  be  available  to 
them  in  real  world  training  environment  (Naikar,  Sanderson,  &  Lintem,  2000). 

WDA  has  been  used  within  DSTO  to  identify  training  functions  for  both  training 
simulators  (Naikar  et  al.,  2000)  and  team  training  (Braithwaite,  2002).  This  latter  study 
used  WDA  to  identify  team  training  functions  for  Combat  Training  Centre  (Live) 
(Braithwaite,  2002). 

The  aim  of  CTC-Live  is  to  provide  collective  training  in  a  realistic  and  challenging 
battlefield  environment  (i.e.,  to  prepare  Combat  Team  and  Battle  Group  as  combined  arms 
teams  for  combat  in  a  realistic  joint  environment  in  order  to  enhance  war  fighting 
operations).  The  study  was  conducted  in  support  for  the  acquisition  of  an  instrumentation 
system  called  CTC  Live  Simulation,  Range  Instrumentation  and  Information  System  (CTC- 
LIS).  WDA  was  used  to  identify  the  training  needs  for  CTC-LIS,  and  then  the  training 
functions  were  analysed  to  identify  the  operational  requirements  for  those  training 
functions.  ° 

Some  of  the  objectives  of  CTC-Live  are  analogous  to  those  for  DMT,  including: 

•  Force  generation  -  increase  unit  readiness  for  deployment  and  war  fighting; 

•  Facilitate  the  evaluation  of  readiness  levels  for  the  combat  force; 
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•  Train  and  evaluate  combat  leaders  and  combat  teams  in  a  realistic  battlefield 
environment; 

•  Provide  a  mission  rehearsal  capability;  Provide  objective  measurements  of 
performance,  feedback,  and  reinforcement  of  lessons  learned,  and; 

•  Contribute  to  the  evaluation  of  operational  concepts. 

The  steps  involved  in  the  study  by  Braithwaite  (2002)  involved  developing  an  abstraction 
hierarchy  of  the  combat  team,  using  that  to  identify  training  functions  for  the  team,  and 
extrapolating  operational  requirements  from  the  training  functions.  This  project  was  able 
to  identify  the  functions  that  must  be  supported  by  the  training  system  design,  rather  than 
technical  or  design  specifications.  The  potential  for  the  use  of  WDA  in  defining  training 
functions  within  the  RAAF  DMT  system  should  be  explored. 
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3.  Theoretical  perspectives  on  learning  and 
instructional  strategies  for  team  training 


There  is  a  vast  literature  on  theories  of  learning  and  instructional  strategies  for  training. 
Training  programs  should  be  informed  by  relevant  theory,  and  hence  an  understanding  of 
the  theoretical  perspectives  is  needed  before  discussing  training  methods  that  have  been 
used.  This  section  of  the  report  is  divided  into  two  distinct  segments.  The  first  segment 
considers  the  prominent  theoretical  perspectives  on  learning  and  skill  acquisition.  The 
second  segment  then  considers  the  range  of  instructional  strategies  for  training. 

3.1  Theoretical  perspectives  on  learning  and  skill  acquisition 

In  order  to  foster  effective  learning,  the  design  and  structure  of  a  training  program  should 
be  informed  by  theory  relating  to  learning  and  skill  acquisition.  The  theory  of  skill 
acquisition  proposed  by  Anderson  (1982)  has  become  prominent  in  the  training  literature. 
The  stages  of  skill  acquisition  described  in  Anderson's  theory  of  cognitive  skill  acquisition 
are  similar  to  those  in  Fitt's  three-phase  theory  (Fitts,  1964,  1965),  although  the 
hypothesised  processes  responsible  for  the  transition  from  novice  to  expert  are  quite 
different  from  those  proposed  by  Fitts.  While  the  majority  of  this  section  will  focus  on 
Anderson's  theory,  it  is  helpful  to  consider  the  earlier  work  of  Fitts. 

Many  reviews  are  in  agreement  that  the  three-phase  theory  proposed  by  Fitts  has  been  the 
most  influential  theory  of  skill  acquisition  in  the  area  of  training  (Annett,  1991;  Patrick, 
1992).  Fitts  (1965)  described  the  development  of  skills  as  the  progression  through  three 
phases:  the  cognitive  phase;  the  fixation  or  associative  phase;  and  the  autonomous  phase. 
The  transition  from  one  phase  to  the  next  was  believed  to  be  a  continuous  process. 

The  cognitive  phase  of  Fitts'  theory  represents  the  initial  cognitive  processes  involved  in 
the  learning  of  a  new  task.  Both  the  trainer  and  trainee  attempt  to  verbalise  what  has  to  be 
learned.  While  expectations  and  procedures  are  explained,  performance  by  the  trainee  is 
error  prone  and  further  guidance  is  required  from  the  trainer.  For  complex  tasks  it  is 
important  that  the  trainee  understands  what  is  involved  in  the  task  before  it  can  be 
attempted.  In  addition  to  learning  rules  and  procedures  governing  what  is  permitted,  the 
trainee  learns  how  a  task  might  be  executed  (i.e.,  develops  understanding).  In  the  second 
associative  phase,  practice  and  the  reduction  of  errors  leads  to  correct  patterns  of 
behaviour  being  established.  In  the  final  autonomous  phase  skilled  performance  becomes 
more  automatic,  requiring  reduced  psychological  resources  such  as  memory  and  attention. 
In  fact,  the  trainee  may  be  unable  to  verbalise  how  the  task  is  performed  suggesting  that 
aspects  of  behaviour  can  be  executed  without  conscious  awareness. 
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The  more  recent  Adaptive  Components  of  Thought  (ACT)  theory  proposed  by  Anderson 
(1982)  develops  Fitts'  concepts  and  explains  how  skill  acquisition  is  a  process  of 
developing  and  refining  production  rules  in  the  transition  from  declarative  to  procedural 
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knowledge.  It  is  assumed  that  experts  have  a  greater  number  of  more  detailed  and 
accurate  production  rules  than  novices.  It  should  also  be  stated  the  declarative  knowledge 
is  factual  knowledge  that  can  be  stated  and  made  explicit,  whereas  procedural  knowledge 
is  implicit  and  often  hard  to  verbalise,  and  is  concerned  with  how  to  perform  a  task  (i.e., 
declarative  knowledge  is  'knowing  that'  whereas  procedural  knowledge  is  'knowing  how' 
-  the  cognitive  basis  for  skilled  performance).  This  theory  has  three  stages  that  are 
outlined  below:  the  declarative  stage;  the  knowledge  compilation  stage;  and  the  tuning 
stage. 

The  first  stage  of  the  ACT  model  describes  when  the  trainee  only  has  limited  declarative 
knowledge  (i.e.,  facts)  about  the  task  and  uses  this  knowledge  in  conjunction  with  general 
problem  solving  procedures  to  attempt  task  performance.  Performance  at  this  stage  often 
relies  on  analogies,  may  make  high  demands  on  attention  and  working  memory,  and  may 
be  slow,  error  prone,  and  effortful.  Anderson  (1982)  also  makes  the  point  that  a  contributor 
to  poor  performance  at  this  stage  often  relates  to  instructions  given  to  the  trainee.  That  is, 
instructions  often  do  not  specify  how  to  perform  that  task.  Procedural  knowledge  will 
develop  as  the  trainee  develops  appropriate  procedures  for  task  performance  through 
practice. 

By  the  end  of  the  second  knowledge  compilation  stage  specific  procedural  knowledge  (via 
developed  production  rules)  has  been  built  up  by  the  trainees.  Knowledge  compilation  is  a 
process  by  which  declarative  knowledge  is  translated  to  procedural  knowledge.  The  first 
process  in  knowledge  compilation  is  composition,  which  is  a  process  by  which  adjacent 
production  rules  are  merged  into  a  single  rule.  Proceduralisation  is  the  second  process 
which  enables  a  production  rule  to  incorporate  more  task  relevant  information.  The  major 
advantage  of  proceduralisation  is  that  information  no  longer  has  to  be  recalled  to  working 
memory  in  order  to  execute  these  rules  (i.e.,  reduced  demands  on  working  memory).  The 
third  and  final  stage  is  the  tuning  stage  in  which  further  improvements  in  skilled 
performance  are  achieved  as  rules  and  skills  are  adjusted  and  fine-tuned. 

Skill  acquisition  is  facilitated  by  the  development  and  refinement  of  production  rules,  and 
this  is  achieved  by  'doing'  the  task  (Anderson,  1982).  Training  therefore  should  identify 
and  pass  on  the  rule  structure  underlying  task  performance  thus  facilitating  the 
development  of  task-specific  procedures.  The  higher  the  quality  of  training  material  the 
lower  the  degree  of  effort  required  by  the  trainee  —  although  no  matter  how  high  quality 
the  training  is,  it  is  critical  that  the  trainee  internalise  the  production  rules  through  practice 
and  modify  them  as  the  task  and  environment  change. 

Like  other  theories  of  skill  acquisition  (e.g.,  Ackerman,  1987;  Ohlsson,  1996),  the  ACT 
theory  assumes  that  learning  has  to  progress  through  a  declarative  stage  before  procedural 
knowledge  can  be  achieved.  As  reviewed  by  Patrick  (1992)  &  Annett  (1991),  there  have 
been  some  researchers  who  have  argued  that  some  knowledge  is  implicit  (cannot  be 
verbalised)  rather  than  explicit,  and  hence  that  implicit  knowledge  may  be  acquired 
differently  from  explicit  knowledge  as  it  does  not  involve  a  transition  from  working 
memory  to  long-term  memory.  Anderson's  theory  however  does  emphasise  that  specific 
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procedures  can  eventually  be  executed  automatically,  thereby  emphasising  a  cognitive 
basis  for  skill  acquisition. 

The  basic  distinctions  made  in  the  ACT  theory  can  also  be  used  to  generate  predictions 
concerning  transfer  (Holding,  1991).  Applying  similar  procedures  to  the  same  declarative 
knowledge  should  lead  to  good  transfer  across  different  tasks,  however  the  same 
declarative  knowledge  may  not  be  very  useful  if  different  procedures  are  required. 
Production  rules  (i.e.,  procedural  knowledge)  can  mediate  transfer  directly  through  their 
application  to  a  range  of  different  cognitive  operations. 

There  are  a  number  of  implications  arising  from  this  model.  The  first  is  that  expertise  is 
domain  specific.  In  dealing  with  content  customary  for  their  work,  experts  use  domain- 
specific  methods  tailored  to  their  field,  rather  than  more  general  problem-solving 
procedures.  In  short,  it  is  how  novices  and  experts  use  their  domain-specific  knowledge 
that  distinguishes  them.  Secondly,  an  expert's  knowledge  is  highly  proceduralised  and 
linked  to  actions  and  conditions  of  applicability.  That  is,  it  is  difficult  to  separate  what  an 
expert  knows  from  how  that  knowledge  is  used  (Means  et  al.,  1993). 

There  are  clearly  some  similarities  in  the  three  stages  of  the  ACT  model  and  those  of  Fitts' 
three-process  model  of  skill  acquisition.  This  declarative  and  procedural  knowledge 
distinction  is  also  the  basis  for  other  theories  and  prominent  models  of  information 
processing  (Meyer  &  Kieras,  199 7).  A  criticism  of  both  theories  is  that  they  are  descriptive 
rather  than  prescriptive  and  as  such  do  not  allow  for  predictions  relating  to  the  duration  of 
the  stages  of  acquisition  of  a  particular  skill  (Patrick,  1992).  Nonetheless,  they  have  been 
prominent  frameworks  in  the  psychological  literature  relating  to  skill  acquisition,  and  they 
provide  a  means  of  conceptualising  the  process  of  skill  acquisition,  and  thereby  provide 
inputs  into  training  design. 

A  more  recent  learning  theory  by  Ohlsson  (1996)  attempts  to  explain  how  people  are  able 
to  detect  and  correct  their  own  performance  errors.  The  basic  principles  of  this  theory  are 
that  errors  are  caused  by  overly  general  knowledge  structures,  that  error-detection 
requires  domain-specific  declarative  knowledge,  that  errors  are  experienced  as  conflicts 
between  what  the  learner  believes  ought  to  be  true  and  what  he  or  she  perceives  to  be  the 
case,  and  that  errors  are  corrected  by  specialising  faulty  knowledge  structures  so  that  they 
become  active  only  in  situations  in  which  they  are  appropriate.  This  theory  is  limited  to 
the  acquisition  of  complex  skills  that  have  significant  cognitive  component. 

Two  major  cognitive  functions  involved  in  learning  from  errors  are  error  detection  and 
error  correction.  With  regard  to  error  detection,  in  order  to  learn  from  an  error  one  must 
be  able  to  detect  it,  or  be  aware  of  the  error.  In  many  training  situations  immediate 
feedback  is  available  via  learning  software  or  tutors.  It  is  less  clear  how  learners  in 
unsupervised  situations  evaluate  their  work  and  are  able  to  detect  errors.  Whilst  not  a  new 
idea  in  psychological  research,  Ohlsson  develops  the  idea  that  errors  are  detected  in 
novices  by  comparing  actual  with  expected  outcomes.  The  emphasis  here  is  on  the 
subjective  view  of  errors  rather  than  the  objective  view  of  errors  (whereby  errors  are 
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deviations  from  the  correct  action).  To  be  informative  during  learning,  errors  must  be 
recognisable  to  the  trainee  before  they  have  attained  mastery  of  the  target  skill. 

Errors  are  recognised  via  particular  features  of  the  situations  they  produce.  Error  signals  is 
the  term  given  to  describe  the  features  or  patterns  of  features  that  indicate  incorrect 
actions.  An  example  of  an  error  feature  would  be  when  a  learner  driver  heard  grinding 
noises  in  the  gearbox  of  a  vehicle.  The  recognition  of  error  signals  requires  domain-specific 
knowledge.  The  learner  must  have  some  knowledge  about  the  range  of  reasonable  actions 
or  responses  in  a  given  situation  in  order  to  recognise  an  action  as  being  incorrect  or 
unreasonable.  As  such,  errors  appear  subjectively  as  conflicts  between  what  the  learner 
believes,  through  prior  knowledge,  should  be  true  and  what  the  learner  perceives  to  be  the 
case. 

Briefly,  Ohlsson  also  discusses  the  distinction  between  practical  and  declarative 
knowledge.  This  distinction  is  necessary  to  account  for  the  fact  that  if  a  learner  has 
sufficient  knowledge  to  recognise  an  action  as  being  incorrect,  then  why  was  that  incorrect 
action  performed  in  the  first  place.  If  an  error  is  started  but  then  recognised  and  corrected, 
then  there  may  be  a  monitoring  mechanism  of  behaviour  that  is  separate  from  the 
mechanisms  responsible  for  the  selection  and  execution  of  the  action  (Norman,  1981).  The 
dissociation  between  execution/action  and  monitoring/ evaluation  can  be  understood  by 
considering  the  distinction  between  practical  and  declarative  knowledge.  The  function  of 
practical  knowledge  is  to  generate  and  organise  action.  The  functions  of  declarative 
knowledge  include  explanation  and  prediction.  Ohlsson  suggests  however  that  declarative 
knowledge  is  prescriptive  rather  than  descriptive,  and  that  the  cognitive  function  that 
supports  it  is  not  explanation  or  prediction,  but  judgement,  which  allows  the  learner  to 
evaluate  the  outcomes  of  their  actions.  So  in  short,  the  function  of  practical  knowledge  is 
to  generate  actions,  and  the  function  of  declarative  knowledge  is  to  support  judgement. 
The  existence  of  two  distinct  knowledge  bases  can  be  used  to  explain  why  a  trainee  can 
perform  an  action  that  they  recognise  as  being  an  error  as  soon  as  they  perceive  the 
outcome. 

With  regard  to  error  correction,  Ohlsson's  theory  describes  displacement  errors,  which  are 
errors  caused  by  overly  general  practical  knowledge.  Initially  a  learner's  behaviour  is 
controlled  by  general  methods  and  rules  that  are  non-restrictive  and  do  not  take  into 
account  the  structure  of  the  particular  task  environment.  These  rules  are  minimally 
restrictive  and  so  will  be  active  in  many  situations  and  therefore  have  a  high  probability  of 
being  active  in  situations  in  which  they  are  not  appropriate.  Individuals  engage  in 
complex  reasoning  (blame  assignment  and  error  attribution)  considering  how  and  why  an 
error  occurred,  and  then  revise  rules,  making  them  more  specialised,  and  limiting  their 
application  to  situations  that  will  produce  desirable  outcomes.  This  hypothesis  has  strong 
similarities  to  the  process  of  rule  revision  discussed  by  Anderson  (Anderson,  1982). 
Ohlsson  concedes  that  these  hypotheses  are  tentative  due  to  a  lack  of  empirical  evidence, 
but  that  a  conceptual  advance  is  that  the  processes  of  error  attribution  and  knowledge 
revision  are  cognitive  functions  that  must  be  specified  to  explain  how  novices  unlearn 
performance  errors. 
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Basically,  we  approach  novel  situations  with  (general)  declarative  knowledge,  and 
through  experience  making  and  correcting  errors,  procedural  knowledge  is  constructed 
(i.e.,  as  production  rules  become  more  specialised  and  modified  they  become  more  specific 
to  a  task  environment).  So  it  is  declarative  not  procedural  knowledge  that  is  transferred  in 
novel  situations.  This  progression  from  declarative  to  procedural  memory  and  the 
development  of  production  rules  is  similar  to  that  discussed  by  Anderson.  With  regard  to 
instruction,  Ohlsson's  theory  would  suggest  that  feedback  be  given  to  the  trainee  which 
should  focus  on  the  trainees  faulty  reasoning  and  the  correct  reasoning,  rather  than  simply 
providing  the  correct  answer.  In  other  words,  feedback  should  focus  on  the  recognition  of 
conditions  and  expectations  which  should  have  been  generated  in  the  situation  in  order  to 
promote  the  transfer  of  higher-order  declarative  knowledge. 

3.1.1  Implications  of  learning  theory  for  measuring  performance  during 
training 

Kraiger  et  al.  (1993)  comprehensively  reviewed  the  literature  concerning  cognitive 
learning  and  drew  inferences  for  training  outcomes.  In  doing  this  they  suggested  three 
general  categories  of  cognitively  based  measures  of  evaluation,  being  verbal  knowledge, 
knowledge  organisation,  and  cognitive  strategies.  These  measures  are  organised,  generally 
speaking,  to  reflect  the  development  and  changing  of  trainee  knowledge. 

Most  theories  of  cognitive  skill  development  state  that  the  acquisition  of  declarative 
knowledge  is  a  necessary  condition  before  acquiring  higher-order  procedural  knowledge 
and  refined  strategies  or  production  rules  (e.g.,  Ackerman,  1988;  Anderson,  1982).  As  such, 
a  measure  of  declarative  knowledge  may  be  more  useful  to  gauge  learning  in  the  early 
stages  of  training.  Declarative  knowledge  can  be  measured  via  multiple-choice  or  free- 
recall  examinations,  however  Kraiger  et  al  (1993)  concede  that  trainee  intelligence  may 
confound  overall  training  success,  i.e.,  more  intelligent  trainees  will  gain  declarative 
knowledge  more  quickly  but  is  this  initial  edge  predictive  of  success  in  subsequent 
training?  The  point  here  is  though  that  tests  of  declarative  (knowing  that)  knowledge  are 
going  to  be  more  value  towards  the  beginning  of  the  training  regime,  and  hence  may  not 
accurately  reflect  higher-order  knowledge  of  more  advanced  trainees  or  those  who  have 
completed  training. 

As  training  progresses  there  is  a  shift  towards  the  acquisition  of  procedural  knowledge, 
which  is  associated  with  the  development  of  knowledge  structures.  As  discussed  in  earlier 
sections,  the  concepts  of  knowledge  organisation  and  mental  models  are  prominent  in  the 
literature.  As  will  be  discussed  in  a  later  section,  researchers  have  been  attempting  to 
develop  reliable  measures  of  mental  models,  and  relating  knowledge  organisation  to  task 
performance. 

Finally,  as  proposed  by  Anderson  (1982),  skill  development  is  a  continuous  process,  and 
so  as  declarative  and  procedural  knowledge  is  developed,  task  strategies  become  more 
refined  and  efficient.  As  discussed  elsewhere,  more  efficient  and  internalised  strategies 
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free  up  cognitive  resources  for  other  executive  functions.  As  discussed  earlier,  the  concept 
of  metacognition  encompasses  the  planning,  monitoring,  and  revision  of  task-oriented 
behaviours,  and  an  understanding  of  the  relationship  between  task  demands  and 
individual  level  of  capability  (Kraiger  et  al.,  1993).  The  measurement  of  metacognitive 
skills  requires  high-level  and  in-depth  examinations  with  subject-matter  experts  to  gain  an 
appreciation  for  the  steps  necessary  to  problem  solve  and  to  progress  towards  an 
objective. 

Having  reviewed  some  of  the  major  theoretical  perspectives  on  learning,  the  following 
sections  discuss  techniques  that  have  been  used  in  the  literature  to  inform  the  design  of 
training  programs. 

3.2  Instructional  strategies  for  team  training 

Instructional  strategies  are  the  techniques  that  are  used  to  develop  and  maintain  required 
competencies.  Instructional  strategies  draw  together  the  tools  for  training  and  the  methods 
of  delivery  to  create  an  environment  for  effective  team  training.  Salas  &  Cannon-Bowers 
(2000)  present  a  review  of  'exemplary'  (instructional)  strategies  that  have  emerged  from 
the  literature.  While  the  Instructional  Systems  Design  (ISD)  approach  &  the  Event  Based 
Approach  to  Training  (EBAT)  seem  to  be  most  appropriate  for  high  technology 
environments,  this  section  of  the  report  initially  focuses  on  reviewing  a  wide  range  of 
approaches  to  team  training. 

Team  training  strategies  result  from  the  combination  of  team  training  tools  and  methods. 
The  tools  of  team  training  include  team  task  analysis,  performance  measures,  feedback, 
task  simulations  and  exercises,  and  principles  of  learning,  training  and  team  performance. 
The  methods  include  information-based  (i.e.,  lectures),  demonstration-based  (i.e., 
videotapes),  and  practice-based  methods  (i.e.,  roleplay  and  simulation).  Guided  by  the 
training  objectives,  the  content  (knowledge,  skills,  attitudes),  tools,  and  methods,  in 
combination,  shape  the  strategies  (Salas  &  Cannon-Bowers,  2000).  Some  of  these  strategies 
are  discussed  briefly  below. 

3.2.1  Team  Coordination  Training 

Team  coordination  training  is  the  most  common  form  of  team  training,  and  is  known  as 
Crew  Resource  Management  (CRM)  training.  Aircrew  Coordination  Training  (more  in 
military  settings),  or  Team  Adaptation  and  Coordination  Training.  While  these 
approaches  have  been  useful  in  aviation,  military,  medical,  and  industrial  settings,  there 
are  some  common  features  despite  the  differing  contexts,  such  as: 

•  Team  coordination  training  is  focussed  on  general  team  training  objectives; 

•  They  are  drawn  from  general  frameworks  of  team  performance  or  group 
dynamics; 

•  They  use  tools  such  as  simulation,  principles,  guidelines  and  specifications,  along 
with  feedback; 
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•  They  combine  information-based  methods  (presentations),  demonstration-based 
(passive  observations  of  behaviour),  and  practice-based  methods  (role  playing  to 
simulation);  and 

•  They  are  usually  short  duration  (2-5  days). 


There  has  been  much  research  aimed  at  the  development  and  evaluation  of  CRM  training. 
CRM  training  is  an  example  of  this  type  of  instructional  strategy  in  commercial  aviation, 
whereby  cockpit  crews  are  trained  on  the  importance  of  teamwork  and  team  performance 
in  an  attempt  enhance  error  management  (Helmreich,  Merritt,  &  Wilhelm,  1999).  The 
claimed  outcome  is  that  crew  members  learn  to  make  better  use  of  the  team  resources, 
with  the  aim  of  minimising  human  error  caused  by  inadequate  coordination  among  teams, 
which  is  particularly  critical  in  emergency  situations  (Wiener,  Kanki,  &  Helmreich,  1993)! 
CRM  training  has  lead  to  a  shift  in  aviation  training  towards  the  social  interactions  within 
the  cockpit,  or  in  other  words,  team  coordination  (Salas,  Prince  et  al.,  1999).  CRM  training 
has  also  been  defined  as  a  family  of  instructional  strategies  that  seek  to  improve  teamwork 
in  the  cockpit  by  applying  well  tested  training  tools  targeted  at  specific  content  (Salas, 
Rhodenizer,  &  Bowers,  2000).  In  accordance  with  Anderson's  ACT  theory,  one  criticism  of 
CRM  training  is  that  because  it  is  predominately  lecture-based,  it  will  effectively  provide 
declarative  knowledge,  but  not  the  transfer  to  procedural  knowledge  that  comes  with 
practice  and  experience  (Salas  et  al.,  2000). 


The  FAA  recommends  that  CRM  training  be  broken  into  three  phases;  a),  an  awareness 
phase  in  which  crew  members  attend  seminars  and  group  exercises  to  learn  the  basics  of 
CRM;  b),  practice  and  feedback  phase  in  which  the  crew  members  fly  realistic  simulator 
scenarios  and  receive  feedback  on  their  performance;  and  c)  a  reinforcement  phase  in 
which  the  concepts  become  part  of  the  organisations  overall  training  and  operational 
practices  (Prince  et  al.,  1992).  The  practice  and  feedback  is  usually  provided  in  the  form  of 
Line  Oriented  Right  Training  (LOFT),  in  which  a  high  fidelity  simulator  is  flown  with  a 
complete  crew  as  if  on  an  actual  line  flight.  All  preparations,  paperwork,  communication, 
and  routine  procedures  that  are  part  of  normal  flight,  are  included  in  the  simulation.  Each 
LOFT  session  would  normally  contain  a  system  failure  or  weather  problem  for  the  crew  to 
resolve.  Crew  performances  are  videotaped  and  then  reviewed  by  the  instructor  and 
trainees. 


While  CRM  has  predominantly  been  researched  and  applied  within  commercial  aviation, 
some  CRM  research  has  concerned  military  aviation  (Salas,  Prince  et  al.,  1999;  Spiker, 
Tourville,  Bragger,  Dowdy,  &  Nullmeyer,  1999;  Thompson,  Tourville,  Spiker,  & 
Nullmeyer,  1999).  Over  nearly  a  decade  of  research,  Salas  et  al  (1999)  attempted  to  study 
CRM  in  the  context  of  Navy  (rotary  wing)  aviation  in  a  systematic  fashion.  CRM  was 
defined  as  a  set  of  teamwork  competencies  that  allow  the  crew  to  cope  with  situation 
demands  that  would  overwhelm  any  individual  crew  member.  These  authors  advocate  an 
EBAT  as  the  best  methodology  for  developing  effective  CRM  training.  EBAT  is  discussed 
in  detail  in  a  later  section.  Their  CRM  instructional  strategy  involved  lectures,  video-taped 
models  of  correct  and  incorrect  behaviours,  and  demonstration  via  opportunities  to 
practice  on  carefully  scripted  scenarios  (as  informed  by  the  EBAT  process),  tracking  of 
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performance,  and  finally  feedback.  They  state  that  the  advantages  of  EBAT  for  CRM 
training  are  that  the  training  program  is  based  on  sound  learning  theory  and  performance 
and  effectiveness  can  be  measured  in  a  systematic  fashion  (Salas,  Prince  et  al.,  1999). 

Few  studies  have  attempted  to  evaluate  the  efficacy  of  CRM  training,  and  this  author  is  yet 
to  see  a  compelling  scientific  validation  of  CRM.  In  one  evaluation  the  skill  areas  of 
communication,  assertiveness,  mission  analysis,  and  SA  were  taught  to  Navy  helicopter 
aircrew  as  part  of  CRM  training  (Salas,  Fowlkes,  Stout,  Milanovich,  &  Prince,  1999).  The 
training  comprised  of  lectures,  video  analysis,  and  practice  and  feedback  through  role- 
play  exercises.  Performance  was  measured  by  the  following: 

•  Trainee  reactions  questionnaire  employed  a  5-point  Likert  scale  to  measure  the 
aircrews'  opinions  on  the  usefulness  of  the  training,  and  to  list  ways  in  which  they 
thought  they  would  use  the  information  provided  to  them  in  training; 

•  Aircrew  Coordination  Attitudes  Questionnaire  (ACAQ)  which  is  a  41-item 
attitude  questionnaire  (that  includes  the  25  items  from  the  Cockpit  Management 
Attitudes  Questionnaire,  which  measures  attitudes  relating  to  communication  and 
coordination,  command  responsibility,  and  recognition  of  stressor  effects); 

•  A  multiple-choice  knowledge  test  to  assess  knowledge  of  concepts  related  to  the 
teamwork  skills  taught  in  training;  and 

•  Behavioural  assessment  using  the  Targeted  Acceptable  Responses  to  Generated 
Events  or  Tasks  (TARGETS)  team  performance  methodology  (discussed  in  more 
detail  in  a  later  section).  Events  were  included  in  an  evaluation  scenario  that 
allowed  for  specific  and  observable  opportunities  for  teams  to  exhibit  team 
coordination  behaviours  (relating  to  the  four  skill  areas  mentioned  above). 

Two  similar  studies  were  conducted  using  Navy  aircrew,  the  major  difference  between  the 
two  studies  being  that  the  second  study  employed  a  repeated  measures  approach  for  the 
behavioural  (TARGETs)  assessment  (Salas,  Fowlkes  et  al.,  1999).  The  scenario  required  the 
team  to  transport  troops  and  cargo  from  a  small  ship  to  an  aircraft  carrier.  The  scenario 
was  produced  using  a  PC  and  Microsoft  flight  simulator.  Performance  was  videotaped 
and  trained  observers  evaluated  performance  according  to  the  TARGETS  approach 
(observers  were  blinded  to  training  condition).  Trainee  reactions  to  CRM  were  positive  in 
relation  to  mission  accomplishment  and  flight  safety,  and  training  increased  the  positive 
attitudes  towards  teamwork  in  the  cockpit  (study  1  only).  Trained  aircrew  also  performed 
significantly  better  on  the  knowledge  test  (related  to  principles  of  teamwork).  With  regard 
to  the  TARGETS,  trained  teams  displayed  more  teamwork  behaviours  than  untrained 
teams  for  high  workload  segments  but  not  lower  workload  segments.  Their  studies 
provide  some  evidence  that  CRM  training  can  improve  teamwork  in  experienced  aviators, 
and  that  the  same  training  can  be  effective  for  both  pilots  and  aircrew. 

Thompson  et  al  (1999)  evaluated  the  effectiveness  of  CRM  training  in  MH-53J  helicopter 
crews  (twin  engine  helicopter).  Sixteen  three-person  crews  participated  in  a  mission 
simulation  during  their  annual  week-long  refresher  training.  The  mission  simulation 
involved  two  other  helicopter  simulations  and  associated  ground  controllers.  A  CRM 
worksheet  was  used  by  observers  to  rate  team  performance  on  a  scale  of  1  to  5  for  the 
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seven  dimensions  of  SA,  crew  coordination,  command  control  and  communication, 
decision  making,  task  management,  mission  evaluation,  and  tactics  employment.  Each 
crew  member  was  given  a  rating  for  each  dimension,  and  then  an  overall  crew  rating  was 
provided.  Mission  performance  was  also  evaluated  by  observers.  Thirty-one  mission 
performance  elements  were  identified  a  priori  and  a  rating  scale  of  1  to  5  was  used  to  rate 
team  performance  for  each  element.  Overall  CRM  and  mission  performance  were  highly 
correlated  (0.84),  and  mission  planning  (and  time  spent  planning)  were  significantly 
correlated  with  mission  performance.  These  behaviours  that  are  associated  with  improved 
mission  performance  should  form  the  basis  for  mission  training  (Thompson  et  al.,  1999). 

While  the  studies  discussed  above  suggest  that  CRM  training  leads  to  improvements  in 
various  dimensions  of  teamwork,  they  are  not  rigorous  scientific  evaluations  of  CRM  and 
the  occurrence  of  human  error  and  accidents. 

3.2.2  Cross  Training 

Cross  training  is  a  relatively  new  strategy  in  team  training  which  has  the  goal  of  providing 
exposure  to  and  practice  with  the  tasks,  roles,  and  responsibilities  of  the  other  team 
members  (Salas  &  Cannon-Bowers,  2000).  It  is  based  on  the  theory  of  the  shared  mental 
model,  in  that  team  members  need  shared  knowledge  structures  (mental  models)  to  be 
able  to  generate  common  predictions  and  expectations,  which  thus  allows  teamwork  to 
occur  without  overt  communication  (Cannon-Bowers  et  al.,  1993).  The  development  of 
such  knowledge  improves  team  members'  ability  to  predict,  anticipate,  and  coordinate 
their  activities  by  increasing  the  accuracy  of  their  knowledge  about  the  roles,  tasks,  and 
information  needs  of  their  team-mates  (Volpe,  Cannon-Bowers,  Salas,  &  Spector,  1996). 
Debrief  sessions  are  designed  to  involve  team  members  in  the  interpretation  of  critical 
events  and  to  help  create  shared  knowledge  of  team-mates'  expectations.  It  has  been 
hypothesised  that  teamwork  that  develops  compatible  mental  models  should  lead  to 
improved  team  performance  (Salas  &  Cannon-Bowers,  2000). 

Kleinman  and  Serfaty  (1989)  suggested  that  that  some  teams  were  able  to  maintain  high 
levels  of  performance  in  high  workload  conditions  because  they  were  able  to  compensate 
for  their  inability  to  communicate  through  the  exercising  of  a  common  understanding  of 
the  situation.  This  knowledge  allowed  the  team  members  to  use  an  implicit  coordination 
strategy  in  which  the  need  for  overt  communication  is  minimised.  This  strategy  also 
emphasises  the  ability  of  effective  team  members  to  anticipate  the  actions  of  other  team 
members  thereby  reducing  the  need  for  overt  communication. 

The  implicit  coordination  that  is  believed  to  allow  teams  to  function  effectively  in  high 
workload  environments  involves  the  use  of  shared  mental  models.  Shared  mental  models, 
in  this  context,  contain  a  common  model  of  the  tactical  situation  among  team  members 
and  mutual  models  of  other  team-member  functions.  It  is  the  latter  type  of  model  that 
allows  team  members  to  pre-empt  the  behaviour  of  others  in  the  team  and  to  coordinate 
actions  in  the  absence  of  explicit  communication. 
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Entin  and  Serfaty  (1999)  used  the  Team  Adaptation  and  Coordination  Training  (TACT) 
procedure  to  measure  the  effects  of  stress  on  team  coordination  strategies.  TACT  involves 
lecture-type  instruction,  observation  via  video  of  correct  and  incorrect  strategies,  and 
practice  via  training  scenarios.  An  additional  procedure  involved  periodic  situation- 
assessments  from  the  instructor  (TACT+).  Six  teams  of  five  naval  officers  performed  in 
either  control,  TACT,  or  TACT+  conditions,  in  both  high  and  low  stress  conditions.  High 
workload  was  induced  via  a  50%  increase  in  the  number  of  targets  in  the  25-30  minute 
scenario.  The  teams  had  to  correctly  infer  the  identity  and  intentions  of  those  targets. 
Trained  naval  officers  rated  team  performance  using  the  Team  Anti-Air  Warfare  (AAW) 
scale,  which  consists  of  12  items  that  assess  AAW  performance  and  15  items  that  assess 
teamwork  dimensions  (team  orientation,  communication,  behaviour,  monitoring 
behaviour,  feedback  behaviour,  back-up  behaviour,  and  coordination  behaviour). 
Psychologists  coded  verbal  behaviour  during  the  sessions  using  a  matrix  that  coded  the 
type  and  content  of  messages,  and  the  person(s)  to  whom  the  message  was  directed. 
Workload  was  measured  using  the  NASA-TLX  measure  of  subjective  workload.  Pre  and 
post-training  tests  showed  that  adaptation  training  via  TACT  and  TACT+  significantly 
increased  process  measures  of  teamwork,  and  the  level  of  teamwork  was  higher  than  for 
the  control  group.  The  adaptation  training,  particularly  TACT+,  allowed  teams  to  manage 
the  effects  of  high  workload.  These  results  combined  suggest  that  team  adaptation  training 
improved  teamwork  and  coordination  strategies,  and  that  teams  can  be  trained  to 
recognise  the  signs  of  increasing  workload.  Unlike  cross-training,  this  training  is  sufficient 
to  maintain  performance  under  stress. 

3.2.3  Team  Self-Correction  Training 

Basically  this  form  of  training  dictates  that  effective  teams  will  review  events,  correct 
errors,  discuss  strategies,  and  plan  for  future  events.  It  attempts  to  formalise  this  natural 
tendency  in  effective  teams.  Knowledge-based  (mental  models),  skill-based,  and  attitude- 
based  competencies  are  developed  without  a  formal  instructor.  Principles  for  this  form  of 
training  are  outlined  elsewhere  (Salas  &  Cannon-Bowers,  2000). 

Team  Dimensional  Training  (TDT)  incorporates  guided  team  self-correction  to  develop 
key  knowledge  and  skills  amongst  team  members  (Smith-Jentsch,  Zeisig,  Acton,  & 
McPherson,  1998).  It  involves  a  facilitator  who  establishes  a  positive  environment,  keeps 
the  team  discussion  focussed,  encourages  and  reinforces  active  participation,  models 
effective  feedback,  and  coaches  team  members  in  stating  their  feedback  in  a  constructive 
manner.  It  is  designed  such  that  teams  can  gain  the  greatest  value  from  self-correction. 
That  is,  it  can  be  used  to  improve  team  performance  by  encouraging  various  types  of 
shared  knowledge,  and  consequently  it  aims  to  develop  shared  mental  models  (Smith- 
Jentsch,  Zeisig  et  al.,  1998). 
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3.3  Approaches  to  team  training  in  complex  high-technology 
environments 

There  is  a  vast  literature  concerning  the  development  of  training  systems  for  complex  and 
technology  rich  environments.  The  systems  approach  identifies  different  subsystems  in  the 
developments  of  training  and  how  they  relate  to  each  other,  and  the  functions  they 
perform.  Instructional  Systems  Development  (ISD)  methods  are  logical  in  nature  in  that 
the  components  are  specified  in  terms  of  the  prescribed  goals.  ISD  methods  are  useful  in 
that  the  development  of  training  is  expressed  as  a  series  of  goals.  While  identifying  what 
the  goals  are,  these  methods  are  not  psychological  in  nature  and  do  not  specify  how  the 
goals  might  be  achieved.  This  latter  process  is  considerably  more  difficult  than  specifying 
the  goals  to  be  achieved  (Patrick,  1992).  There  are  literally  hundreds  of  different  ISD 
methods.  All  tend  to  involve  broadly  similar  stages.  Further  detail  of  an  ISD  process  for 
complex  environments  is  presented  below. 

3.3.1  The  Instructional  Systems  Development  Process 

Following  is  brief  discussion  of  some  more  specific  tools  for  team  training  associated  with 
high  technology  environments.  While  not  specifically  related  to  teams,  Williams  &  Pearce 
(1999)  outline  the  ISD  process  for  Virtual  Environment  (VE)  training  technologies. 

Needs  Analysis:  A  Needs  Analysis  is  required  when  there  is  a  change  to  operational 
procedures,  the  development  of  new  processes,  the  desire  to  cut  costs,  and  so  on.  Basically 
this  process  involves  identifying  the  tasks  involved  to  perform  the  activity.  An  example 
provided  by  Williams  &  Pearce  (1999)  concerns  the  US  Navy  and  a  decision  to  downsize 
and  seek  alternative  and  lower  cost  methods  for  training.  It  was  decided  that  full  mission 
simulation -based  training  was  a  potential  means  to  reduce  training  costs.  So  the  tasks 
involved  in  ship  handling  performed  as  part  of  the  job  were  identified  to  determine 
alternative  training  technologies  and  techniques  that  might  be  available.  It  is  then 
necessary  to  determine  which  objectives  critical  to  meeting  the  organisational  objectives 
are  involved  in  the  current  job  (i.e.,  how  important  is  the  job  relative  to  the  organisational 
objectives).  A  major  goal  of  the  Needs  Analysis  is  to  specify  the  job  requirements  and/or 
objectives. 


In  fact,  when  the  instructor  station  is  developed,  even  conceptually,  to  a  given  point  an 
KD  or  training  needs  analysis  can  be  done  to  identify  the  tasks  to  performed  by  the 
instructor,  the  expected  sequence  of  tasks,  potential  conflicts  between  instructional 
features,  the  areas  in  which  the  instructors  will  need  training,  and  the  nature  of 
coordination  between  instructors  when  more  than  one  instructor  is  involved  (Cream, 
Eggemeier,  &  Klein,  1978).  Furthermore,  each  feature  of  the  instructor  station  should  be 
evaluated  against  a  number  of  dimensions:  demands  placed  upon  instructor  time  and 
attention;  anticipated  frequency  of  use  of  the  instructional  feature;  computational 
demands,  location  of  the  instructor;  and  actual  training  support  provided  by  the  feature 
(Cream  et  al.,  1978).  As  with  individual  tasks  in  a  training  task  functional  analysis,  each  of 
the  instructional  features  should  also  be  assessed  by  SMEs  along  the  dimensions  of 
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criticality  (determination  of  the  training  implications  of  not  including  that  feature), 
frequency  of  performance  (the  expected  use  of  the  feature),  and  difficulty  of  performance 
(difficulty  of  operation  of  the  feature). 

Quite  surprisingly,  consideration  of  the  training  requirements  for  instructor  stations  are 
infrequently  discussed  in  the  team  training  literature.  Clearly  however  the  role  of  the 
instructor  is  critical  in  the  success  of  a  team-training  program.  There  are  many  stages  of 
the  training  system  development  process  that  can  influence  the  instructor  station 
requirements  for  team  training.  For  example,  determination  of  the  performance 
measurement  capabilities  can  influence  design  of  the  instructor  station  because  the 
instructor  must  be  able  to  observe  and  collect  the  required  performance  data  (Cream  et  al., 
1978).  The  role  of  the  instructor  is  one  primarily  of  instruction  rather  than  evaluation,  and 
the  instructor  station  should  support  that  performance  of  a  number  of  instructor  functions: 
controlling  and  setting  up  tasks;  measuring  trainee  performance  on  the  tasks;  displaying 
and  recording  performance  measurements  in  a  useful  format;  and  presenting  the 
measurements  and  other  instructional  communications  as  feedback  to  the  trainees  (Cream 
et  al.,  1978). 

Task  Analysis:  Having  specified  the  job  objectives  from  the  needs  analysis,  the  next  step  of 
the  ISD  process  is  to  identify  the  tasks  that  must  be  performed  by  teams  or  individuals  in 
order  to  meet  the  job  objectives,  which  is  achieved  using  task  analysis.  Each  of  the  job 
objectives  must  be  defined  in  terms  of  the  specific  functions  which  must  be  performed  by 
each  person.  Task  analysis  therefore  involves  breaking  down  each  job  objective  into  the 
component  tasks  required  to  achieve  these  objectives.  Typically  the  task  analysis  will 
involve  a  person  observing  an  individual  performing  the  job  activities  necessary  to  achieve 
the  job  objectives. 

The  activities  to  be  performed  for  each  particular  task  are  analysed  in  terms  of  the  stimuli 
in  the  task  environment  and  the  responses  to  those  stimuli.  The  analyst  can  then  infer  the 
skills  and  knowledge  required  to  perform  those  tasks.  The  input  characteristics  must  be  set 
beforehand  by  the  analyst.  That  is,  the  baseline  level  of  skills  and  knowledge  possessed  by 
the  operator  (i.e.,  degree  of  existing  expertise,  knowledge,  and  skill).  In  addition,  the 
analyst  would  also  specify  the  fundamental  abilities,  such  as  visual  and  auditory 
requirements,  physical  requirements,  etc,  related  to  successful  completion  of  the  job  tasks. 

Finally,  having  documented  the  skills,  knowledge,  and  abilities  necessary  to  perform  the 
task,  the  analyst  should  then  specify  indexes  that  can  be  used  to  measure  performance  on 
the  various  tasks.  Consequently  the  analyst,  in  consultation  with  experts,  can  derive  an 
agreed-upon  level  of  performance  that  would  be  indicative  of  someone  who  had  learned 
the  tasks  with  a  level  of  mastery.  These  performance  measures  can  be  used  to  measure  the 
progress  of  a  trainee  through  a  training  program.  The  outputs  from  task  analysis  are 
clearly  critical  to  the  design  of  a  training  system. 

Relatively  little  work  has  focussed  on  task  analysis  at  the  team  level.  As  such,  designers  of 
team-training  programs  have  often  been  required  to  decide  which  tasks  should  be  trained 
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individually  and  which  as  a  team  without  systematic  procedures  or  guidance.  There  needs 
to  be  a  clear  distinction  between  individual  and  team  tasks  for  team  training  to  be  fully 
effective.  There  have  been  few  attempts  to  develop  a  team  task  analysis  tool,  although 
Cannon-Bowers  and  colleagues  have  developed  a  methodology  for  specifying  the 
coordination  demands  of  complex  team  tasks.  This  methodology  has  been  useful  in  the 
aviation  domain  for  specifying  tasks  that  require  coordination  and  team  training 
objectives.  It  extends  upon  traditional  task  analysis  by  asking  experts  to  provide 
information  about  task  interdependency  (Salas  &  Cannon-Bowers,  2000). 

Curriculum  Development:  The  outputs  from  the  task  analysis  are  then  used  to  develop  the 
lessons  or  scenarios,  the  objectives  of  each  scenario,  and  the  critical  levels  of  performance 
that  must  be  achieved. 

Selection  of  a  Delivery  Device  and  Media:  Basically  system  designers  must  analyse  the 
curriculum  material  and  determine  the  most  appropriate  method  of  presentation  for  the 
training  program.  Cost  can  be  a  significant  issue  here,  and  hence  cost-benefit  analyses  can 
be  conducted  at  this  stage. 

Training  Program  Conduct  and  Evaluation:  This  section  briefly  discusses  how  to  examine 
whether  a  training  program  is  effective.  That  is,  conduct  an  experiment  in  which  a  control 
group  does  not  receive  training  and  an  experimental  group  that  does  receive  the  training, 
and  examine  differences  in  pre  and  post  test  performance.  If  the  training  program  is 
effective  then  significant  differences  between  pre  and  post  test  performance  should  be 
found  (as  a  function  of  the  training  program). 

Feedback  Cycle:  This  relates  to  consideration  of  the  evaluation  and  the  costs  of  training 
versus  the  improved  performance. 

It  is  important  to  bear  in  mind  that  the  emergence  of  new  technologies  does  not  justify 
their  use  for  training  unless  the  effectiveness  of  training  can  be  assured. 

3.3.2  Event-Based  Approach  to  Training  (EBAT) 

A  related  but  quite  distinct  approach  to  training  design  is  EBAT.  This  technique  has  been 
used  widely  in  recent  years,  particularly  in  military  environments.  It  is  based  on  the 
premise  that  learning  is  a  cognitive  and  behavioural  process.  Effective  performance  often 
requires  a  range  of  competencies  across  knowledge,  skills,  and  attitudes,  as  discussed  in 
the  previous  section.  Effective  performance  can  become  more  challenging  when  few 
procedures  exist  or  when  more  than  one  approach  can  be  used  to  execute  the  task. 
Learning  must  therefore  focus  on  cognitive  and  behavioural  processes,  and  the  training 
systems  must  provide  an  environment  that  supports  the  acquisition  of  the  competencies 
required  to  achieve  task  performance  (Oser,  Cannon-Bowers,  Salas,  &  Dwyer,  1999;  Salas, 
Oser,  Cannon-Bowers,  &  Daskarolis-Kring,  2002). 
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A  systematic  approach  to  learning  will  facilitate  skill  acquisition  &  retention.  Complex 
skill  acquisition  requires  a  systematic  approach  to  learning  that  is  based  on  sound  learning 
and  measurement  methods,  strategies,  and  tools.  This  systematic  approach  must  contain 
specific  pre-planned  instances  where  the  trainees  demonstrate  their  ability  to  perform  the 
tasks  and  receive  feedback  on  critical  competencies.  These  instances  must  be  believable 
and  transparent  to  the  trainee  otherwise  the  task  may  not  be  performed  realistically  (Oser 
et  al.,  1999). 

Training  for  complex  environments  requires  a  scenario-based  approach.  Effective  learning 
environments  need  to  facilitate  the  ability  to  develop  and  maintain  competencies  (i.e., 
knowledge,  skills,  and  attitudes)  required  to  perform  the  task.  Scenario-based  (or  event- 
based)  training  can  ensure  that  such  learning  occurs  by  tightly  linking  critical  tasks, 
objectives,  scenario  design  and  execution,  measurement,  and  feedback  (Dwyer,  Oser, 
Salas,  &  Fowlkes,  1999;  Oser  et  al.,  1999). 

The  tactical  decision-making  involved  in  distributed  training  scenarios  necessitates  that 
events  (cues)  are  embedded  in  the  task  at  different  time  intervals.  Sets  of  events  become 
the  basis  for  assessment  because  not  all  aspects  of  performance  can  be  measured  during  an 
exercise.  These  events  are  there  to  provide  opportunities  for  the  required  team-based 
competencies  to  be  exhibited  and  subsequently  measured  and  used  for  providing 
feedback  (Salas  &  Cannon-Bowers,  2000). 

While  this  form  of  training  has  been  used  in  aviation  settings,  tactical  team  decision¬ 
making  environments,  and  distributed  training  scenarios,  it  has  great  potential  for  further 
military  applications  (e.g.,  combat  crews  can  be  trained  and  systematically  exposed  to 
events  that  represent  the  major  types  of  threat  situations).  Variations  in  simulation  fidelity 
can  be  used  to  allow  crews  to  practice  critical  decision-making  and  coordination  skills  in 
response  to  particular  (important)  cues.  The  provision  of  prompt  and  meaningful  feedback 
to  physically  separated  trainees  is  a  focus  of  current/ future  research  (Salas  &  Cannon- 
Bowers,  2000). 

EBAT  is  used  to  design  simulation-based  exercises,  and  these  techniques  create  training 
opportunities  by  systematically  identifying  and  introducing  events  within  training 
exercises  that  provide  known  opportunities  to  observe  specific  behaviours  of  interest 
(Fowlkes,  Dwyer,  Oser,  &  Salas,  1998).  EBAT  has  been  used  widely  in  the  team  training  in 
complex  environments  literature,  such  as  for  evaluating  tactical  decision-making 
performance  of  AAW  CIC  teams  0ohnston  et  al.,  1997),  and  for  aircrew  coordination 
training  (Fowlkes  et  al.,  (1994). 

In  EBAT,  the  training  exercise  is  the  curriculum,  and  the  primary  goal  is  to  provide 
trainees  with  opportunities  to  develop  critical  competencies  by  receiving  practice  in 
simulated  environments  that  are  representative  of  actual  operational  conditions,  and 
receiving  feedback  linked  to  specific  events  that  occur  during  training.  EBAT  highlights 
linkages  between  all  phases  of  training  (design,  measurement,  feedback)  and  so  can 
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inform  the  design  of  system  infrastructures  (Oser  et  al.,  1999).  The  components  of  EBAT 
are  described  in  more  detail  below. 

The  EBAT  framework  highlights  linkages  between  all  phases  of  training.  The  phases  will 
be  discussed  briefly  in  the  following  sections.  EBAT  begins  with  the  specification  of  the 
training  objectives,  including  the  critical  tasks  that  should  be  performed,  the  conditions 
under  which  those  tasks  should  be  performed,  and  the  standards  of  acceptable 
performance  (Dwyer  et  al.,  1999).  The  key  phases  of  EBAT  are  discussed  below. 

l:...Skill  Inv entory / Historical  Performance  Data:  Scenarios  should  be  based  upon  the 
training  audience's  inventory  of  skills  and  archived  historical  performance.  This  process 
should  highlight  areas  of  skill  that  need  to  be  improved,  rather  than  maintained,  and 
should  result  in  a  more  focussed  and  relevant  training  program  (Oser  et  al.,  1999). 

2.  Task  Lists:  Specific  tasks  can  be  identified  to  inform  scenario  design  based  upon  the  skill 
inventory  and  historical  performance  data.  Ideally  an  organisation  would  have  detailed 
lists  of  tasks,  techniques,  and  approaches,  that  must  be  performed  for  effective  operation 
of  the  complex  system. 

3-.Leaming  Objectives/Competencies:  Objectives  and  related  competencies  associated 
with  the  skill  inventories,  historical  data,  and  task  lists,  can  then  be  specified.  Objectives 
can  be  task  specific  or  refer  to  general  competencies  across  tasks  (e.g.,  SA,  decision¬ 
making).  The  learning  objectives  should  be  specified  in  such  a  way  as  to  enable  objective 
and  subjective  assessment  about  whether  they  were  achieved  (Oser  et  al.,  1999). 

The  learning  objectives  specify  the  training  areas  on  which  to  focus.  They  represent 
components  of  behaviour  that  may  have  been  deficient  in  the  past,  difficult  to  perform,  etc, 
and  so  must  be  practiced  in  order  to  maintain  proficiency  (Dwyer  et  al.,  1999). 

4.  Scenario  Events/Scripts:  A  key  distinguishing  feature  of  EBAT  is  the  scenario  events 
which  provide  specific  opportunities  for  the  training  audience  to  practice  critical  tasks  and 
competencies  associated  with  the  learning  objectives.  The  events  allow  the  trainees  to 
demonstrate  their  strengths  and  weaknesses  for  the  purposes  of  performance 
measurement  and  feedback.  The  Master  Scenario  Event  List  (MSEL)  is  an  organised 
structure  that  lists  all  of  the  events  that  are  planned  for  a  particular  scenario,  and  contains 
timing  of  events,  relationships  between  events  and  objectives,  etc  (Oser  et  al.,  1999). 

EBAT  emphasises  the  requirement  for  events  to  be  driven  by  training  objectives.  Another 
aspect  of  these  events  is  that  sequential  dependencies  are  minimised,  or  in  other  words, 
much  effort  is  devoted  to  ensuring  that  the  ability  to  respond  to  a  given  event  is 
independent  of  performance  on  the  previous  event  (Fowlkes  et  al.,  1994). 

To  ensure  task  consistency  across  teams,  scenario  scripts  are  developed  that  detail  when 
events  should  occur  and  the  communications  that  should  occur  from  other  personnel  as 
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part  of  the  scenario.  These  scripts  also  allow  the  instructor  to  anticipate  events  and  to 
evaluate  performance  based  on  the  behaviours  of  interest  (Fowlkes  et  al.,  1994). 

The  trigger  events  are  identified  for  each  learning  objective,  and  should  create  specific 
opportunities  for  the  trainees  to  demonstrate  their  ability  to  perform  the  tasks  associated 
with  the  learning  objectives  (Dwyer  et  al.,  1999).  These  events  also  provide  controlled 
situations  in  which  performance  can  be  measured.  The  use  of  controlled  trigger  events  is 
especially  critical  for  non-routine  tasks  when  the  opportunity  for  practice  may  not  arise 
unless  intentionally  introduced.  An  example  used  by  Dwyer  et  al.  (1999)  is  a  learning 
objective  that  focuses  on  the  proper  execution  of  procedures  for  switching  to  a  back-up 
communications  plan. 

5.  Performance  Measures/Standards:  The  development  of  performance  measures  and 
tools  required  for  data  collection  associated  with  the  trigger  events  is  the  next  step  in 
EBAT.  Data  should  also  be  collected  on  both  outcomes  and  processes  (Oser  et  al.,  1999). 

Multiple  events  of  varying  difficultly  should  be  specified  for  each  learning  objective,  and 
introduced  throughout  the  training  exercise,  to  allow  the  development  of  a  training  profile 
of  how  performance  is  going  for  each  objective  across  a  number  of  conditions  (Dwyer  et 
al.,  1999).  There  is  considerable  discussion  about  performance  measurement  in  a  later 
section  (Section  4). 

6.  Performance  Diagnosis:  Basically  this  states  that  the  diagnosis  of  data  is  a  critical  stage 
of  the  training  process.  The  diagnosis  can  be  via  automated,  semi-automated,  or 
instructor-led  feedback. 

7.  Feedback  and  Debrief:  Feedback  is  linked  to  the  performance  measures,  which  is  linked 
to  the  trigger  events  and  the  learning  objectives  -  again  emphasising  how  EBAT  links  the 
critical  scenario  components.  Feedback  should  relate  to  both  process  and  outcome  (Oser  et 
al.,  1999).  The  EBAT  approach  emphasises  the  need  for  data  to  be  presented  in  a  format 
that  supports  feedback  to  trainees  (Dwyer  et  al.,  1999). 

In  summary,  a  training  objective  is  identified,  from  which  learning  objective(s)  can  be 
defined.  Ideally  more  than  one  trigger  event  will  be  used  to  assess  performance  for  each  of 
the  learning  objectives,  and  performance  measures  are  used  for  feedback  in  the  after¬ 
action  review  (Dwyer  et  al.,  1999).  Table  2  below  illustrates  the  link  between  training 
objective  and  performance  for  a  close  air  support  task. 
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Table  2:  An  example  to  illustrate  the  link  between  training  objectives  and  performance 
measurement  using  EBAT  (adapted from  Dwyer  et  al,  1999). 


Training 

Objective 

Learning 

Objective 

Trigger  Event 

Performance 

Measures 

Feedback 

-  Control  close 
air  support 
attack  task 

-  Daytime 
operations 

-  No  errors 

-  Demonstrate 
behaviours 
required  to 
adapt  to  loss  of 
communication 
during  air  attack 

_ _ _ _ 

-  Introduce 

primary 
communication 
failure  at  specific 
time  in  exercise 

-  Pilots  notified 
of 

communication 

loss 

-  Consideration 
of  other 

resources 

-  Back-up  plan 
implemented 

-  Replay  event 

-  Provide  data 

-  Discuss  what 
occurred,  why, 
and  how  to 
remedy 

Several  tools  can  be  used  to  measure  the  aspects  of  performance  identified  above, 
including  TARGETS.  Measures  of  performance  are  discussed  in  detail  in  a  later  section. 

It  is  claimed  that  there  are  at  least  two  ways  that  EBAT  can  be  used  for  cognitive  learning 
and  the  development  of  knowledge  structures.  Firstly,  an  EBAT  that  is  driven  by  CTA  can 
expose  trainees  to  known  cue  situations  for  comparison.  An  example  provided  by  Fowlkes 
et  al  (1994)  is  that  a  controller  can  learn  to  distinguish  between  situations  that  require  a  call 
for  descriptive  information  (e.g.,  bogey  capping,  high)  versus  those  that  require  a  call  for 
directive  information  (e.g.,  hornet  21,  break  right).  Hence,  it  is  claimed,  the  development  of 
schema  can  be  fostered  through  careful  design  of  training  scenarios.  Secondly,  EBAT  can 
also  be  used  to  measure  trainees  procedural  (how  to)  knowledge,  and  thus  can  be  used  in 
the  assessment  of  cognitive  skills  (Fowlkes  et  al.,  1994). 

3.3.2. 1  How  does  EBAT  differ  from  other  frameworks  for  training  design? 

Unlike  approaches  such  as  the  ISD  approach,  EBAT  specifically  focuses  on  training  where 
the  trainee  is  immersed  in  simulated  operational  environments.  This  has  allowed  EBAT  to 
address  the  unique  aspects  of  training  in  simulated  environments  where  the  scenario  is  the 
curriculum.  EBAT  has  also  been  applied  extensively  in  training  situations  that  incorporate 
simulation.  Finally,  EBAT  results  in  quantifiable  improvements  involving  simulation  and 
both  individual  and  team  performance  of  complex  tasks  (Oser  et  al.,  1999). 

Fowlkes  et  al  (1998)  summarise  the  key  training  and  measurement  benefits  of  the  EBAT. 
The  event-based  scenario  design  means  that  the  training  exercises  are  linked  directly  with 
the  training  requirements.  This  is  the  case  because  the  task  lists,  required  competencies, 
etc,  feed  directly  into  the  design  of  the  specific  scenarios.  The  level  of  control  over  scenario 
content  allows  for  reliable  observation  of  the  targeted  competencies  and  accurate 
assessment  of  those  competencies.  Similarly,  the  event-based  observation  allows  for 
standardisation  of  observations  and  control  over  task  content.  Finally,  EBAT  reduces 
instructor  workload  as  the  requirements  for  real-time  observations  are  reduced.  EBAT 
prompts  specific  behaviours  to  occur  and  thus  allows  them  to  be  learnt  and  reinforced. 
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33.2.2  Specific  Event-Based  Learning  Methods 

An  example  of  a  widely  used  EBAT  method  is  the  TARGETS,  which  involves  in-depth 
analyses  of  specific  simulator  scenarios  that  are  used  to  evaluate  training  effectiveness 
(Fowlkes  et  al.,  1994).  TARGETS  was  developed  to  evaluate  the  behavioural  component  of 
a  Navy  skill-based  aircrew  coordination  program.  Detailed  examples  using  the  TARGETS 
methodology  are  discussed  in  the  next  section  of  the  report  concerned  with  the 
measurement  of  team  performance. 

A  second  method  will  be  discussed  in  more  detail  here.  The  TADMUS  program  (Tactical 
Decision  Making  Under  Stress)  has  been  used  as  the  foundation  for  an  event-based 
learning  method  for  designing  a  continuous  training  cycle  (Stretton  et  al.,  1999).  This  six 
year  program  of  research  was  instigated  after  an  incident  in  1988  involving  the  shooting 
down  of  an  Iranian  airliner  by  a  US  Navy  vessel.  The  goal  of  TADMUS  was  to  improve 
methods  to  train  tactical  decision  making  in  teams  and  individuals  in  stressful  situations 
such  that  proficiency  is  enhanced  (Cannon-Bowers  &  Salas,  1998).  This  was  done  within 
the  context  of  a  ship's  Combat  Information  Centre  (CIC)  environment. 

The  major  theoretical  perspectives  that  were  drawn  upon  in  the  development  of  the 
TADMUS  program  were  naturalistic  decision-making  and  shared  mental  models 
(Cannon-Bowers  &  Salas,  1998).  After  consideration  of  theoretical  perspectives  it  was 
possible  to  generate  a  series  of  hypotheses  for  training.  Some  of  the  training  interventions 
derived  from  the  hypotheses  included  EBAT,  TACT,  team  leader  training,  and  cross 
training  (Cannon-Bowers  &  Salas,  1998).  Team  leader  training  focussed  on  improving  the 
feedback  skills  of  the  people  who  led  the  team's  prebrief  and  debrief  associated  with  each 
exercise.  It  included  lectures,  demonstrations  of  effective  and  ineffective  briefings,  and 
roleplays  (Smith-Jentsch,  Johnston,  &  Payne,  1998). 

After  an  analysis  of  over  50  videotaped  teams  in  the  CIC  environment,  SMEs  identified  a 
number  of  teamwork  behaviours  that  seemed  to  differentiate  high  and  low  performing 
teams  (Johnston  et  al.,  1997).  These  behaviours  were  grouped  under  the  dimensions  of  SA, 
communication,  supporting  behaviour,  and  team  initiative  or  leadership.  The  Anti-Air 
Teamwork  Observation  Measure  (ATOM)  was  subsequently  developed,  and  records 
observations  of  specific  behaviours  associated  with  these  dimensions.  Performance 
measurement  techniques  are  discussed  in  the  next  section  of  the  report. 

3.3.3  Summary 

Only  the  more  prominent  and  relevant  forms  of  team  training  and  instructional  strategies 
have  been  reviewed  here.  While  the  EBAT  approach  to  team  training  in  complex  high 
technology  environments  has  been  widely  used  in  the  military  domain,  there  may  also  be 
a  role  for  other  forms  of  team  training,  such  as  team  coordination  training  and  cross 
training.  The  extent  to  which  these  other  forms  of  team  training  might  be  involved  in 
distributed  training  would  depend  largely  on  the  extent  to  which  the  critical  team 
processes  (SA,  mental  models,  etc)  are  adequately  covered  in  the  EBAT  approach.  The 
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following  section  discusses  literature  concerning  the  measurement  of  these  team 
processes. 
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4.  Measurement  of  team  processes 

There  are  several  means  by  which  to  measure  the  effectiveness  of  a  training  simulation. 
These  methods  include  utility  evaluations  (seeking  opinions  of  effectiveness  from  relevant 
sources),  in-simulator  learning  (evaluating  improvements  in  simulator  performance),  and 
transfer  of  training  studies  (examining  if  and  how  simulation  training  affects  subsequent 
performance)  (Bell  &  Waag,  1998).  Across  many  fields  including  medical,  nuclear  power 
plant,  commercial  aviation,  and  military  aviation,  there  are  few  robust  empirical 
evaluations  of  the  effectiveness  of  simulator-based  training  (Bell  &  Waag,  1998;  Thurman 
&  Dunlap,  1999).  The  majority  of  this  section  is  therefore  concerned  with  performance 
before,  during,  and  after  a  training  program.  Few  transfer  of  training  studies  have  been 
conducted,  and  are  discussed  towards  to  the  end  of  this  section. 

Measurement  of  performance  is  critical  for  effective  learning.  Performance  measurement 
should  include  collection  of  data  pertaining  to  the  outcomes  (whether  the  right  decision 
was  made)  and  the  processes  (whether  the  decision  was  made  correctly).  A  range  of 
performance  measures  and  systematic  assessments  of  those  measures  ensure  the  effective 
provision  of  feedback,  and  subsequent  improvements  in  performance.  Performance 
measurement  is  therefore  critical  in  ensuring  that  the  training  program  is  having  its 
intended  effect,  and  allows  trainers  to  assess,  diagnose,  and  refine  performance  (Oser  et 
al.,  1999).  In  the  team  training  scenario  the  performance  measures  need  to  capture  the 
dynamic  and  multilevel  nature  of  teamwork,  which  is  not  a  trivial  task.  Salas  and  Cannon- 
Bowers  (2000)  state  that  any  measures  of  team  performance  must  have  the  following 
features;  be  theoretically  based;  assess  and  diagnose  team  performance;  address  team 
outcomes  as  well  as  processes;  capture  moment-to-moment  behaviours,  cognitions, 
attitudes,  and  strategies;  consider  multiple  levels  of  analysis;  consider  multiple  methods  of 
measurement;  provide  a  basis  for  remediation,  and;  rely  on  observation  by  experts. 

Dwyer  et  al  (1999)  discuss  the  importance  of  using  both  outcome  measures  and  process 
measures  of  team  performance.  With  regard  to  outcome  measures,  tools  have  been 
developed  to  measure  data  such  as  the  number  of  bombs  released,  the  timing  of  bomb 
releases,  and  the  percentage  of  bombs  resulting  in  intended  impact.  Outcome  measures 
provide  information  to  identify  what  happened  in  an  exercise  and  to  help  to  identify  areas 
of  trainee  difficulty.  Process  measures  focus  on  behaviours  as  the  exercise  unfolds  and  are 
used  to  determine  why  the  difficulties  occurred.  It  is  important  to  measure  team  processes 
rather  than  simple  team  outcome  measures  because  process  measurement  is  important  for 
providing  information  that  is  useful  for  diagnosing  the  cause  of  poor  performance  (Paris, 
Salas,  &  Cannon-Bowers,  1999). 

The  challenge  in  measuring  team  performance  is  to  define  good  measures  of  team 
processes.  As  Salas  et  al  (1995)  review,  a  number  of  measurement  techniques  have  been 
developed  for  assessing  military  team  performance.  These  measures,  such  as  the  Critical 
Team  Behavior  Form  (Morgan  et  al.,  1986),  the  Aircrew  Coordination  Observation  and 
Evaluation  form  (Franz  et  al.,  1990),  the  Teamwork  Observation  Measure,  the  Teamwork 
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Rating  Scale  (Brannick,  Roach,  &  Salas,  1993),  and  the  Anti- Air  Warfare  Team  Performance 
Index  (Johnston  et  al.,  1997),  all  involve  having  expert  observers  rate  the  teams,  either  on¬ 
line  or  via  analysis  of  videotaped  performance,  according  to  a  predetermined  checklist  of 
teamwork  behaviours.  Discussions  following  focus  on  process  measures  of  performance. 

Before  discussing  in  detail  some  of  the  process  measures  of  team  performance,  it  is 
necessary  to  state  that  it  is  quite  difficult  to  interpret  these  findings  with  a  high  degree  of 
clarity.  As  discussed  in  an  earlier  section  of  this  report,  there  are  a  number  of  higher-order 
cognitive  processes  for  teams  (e.g.,  control  of  attention,  shared  SA,  shared  mental  model, 
and  metacognition).  There  are  also  several  core  dimensions  of  teamwork  that  have  been 
identified  in  the  literature  (e.g.,  communication,  team  orientation,  team  leadership, 
monitoring  and  awareness  of  the  activities  and  performance  of  other  team  members, 
feedback,  and  coordination).  The  majority  of  studies  that  are  discussed  in  the  following 
section  appear  to  use  at  least  one  of  these  teamwork  dimensions  in  order  to  make 
inferences  about  team  processes.  For  example,  studies  will  often  measure  the  dimension  of 
communication  in  order  to  make  inferences  about  the  team's  SA,  or  measure  coordination 
to  make  inferences  about  shared  mental  models.  The  difficulty  in  interpreting  the  results 
from  previous  studies  is  that  each  study  seems  to  use  different  combinations  of  teamwork 
dimensions  to  draw  conclusions  about  team  processes.  This  is  apparent  via  quick 
inspection  of  Table  3  which  summarises  some  of  the  team  dimensions  and  processes  that 
have  been  examined  in  the  literature.  Given  the  range  of  teamwork  dimensions  used  in  the 
literature  it  would  seem  that  there  is  not  solid  agreement  about  the  critical  teamwork 
dimensions  for  optimal  team  performance. 

This  next  sections  (4.1  &  4.2)  describe  studies  that  have  measured  dimensions  of  teamwork 
such  as:  communication;  team  coordination;  team  adaptation;  information  exchange; 
supporting  behaviour;  team  leadership;  assertiveness;  decision-making;  technical 
coordination;  interpersonal  cooperation;  team  spirit,  and;  cross-monitoring.  The  section  on 
Team  Knowledge  (4.2.1)  focuses  more  on  the  measurement  of  Team  SA  and  shared  mental 
models. 
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4.1  Measurement  using  the  TARGETS  methodology 

As  discussed  earlier,  the  TARGETS  methodology  involves  in-depth  analyses  of  specific 
simulator  scenarios  that  are  used  to  evaluate  training  effectiveness  (see  Fowlkes  et  al., 
1994).  The  TARGETS  methodology  was  developed  within  the  context  of  the  EBAT.  It  uses 
a  behaviourally  focused  checklist  to  record  observations  of  team  behavior.  SMEs,  doctrine, 
manuals,  and  standard  operating  procedures,  are  used  to  define  anticipated  behaviours 
for  specific  events  a  priori.  The  behaviours  are  listed  in  the  checklist  in  the  approximate 
order  in  which  they  are  likely  to  occur  during  the  exercise,  and  are  further  broken  into 
discrete  phases  of  the  mission.  During  an  exercise  observers  rate  each  behaviour  on  the 
checklist  as  acceptable,  unacceptable,  or  not  observed.  In  terms  of  validity,  TARGETS 
checklists  have  been  shown  to  have  good  inter-observer  reliability  (r=0.94)  and  internal 
consistency  (r=0.81),  and  have  good  face  validity  for  observers  and  participants  (Fowlkes 
et  al.,  1994).  The  data  can  also  be  used  to  provide  more  detailed  feedback  to  trainees  than 
outcome  data.  The  groups  of  behaviours  in  the  checklist  can  be  divided  into  functionally 
related  clusters  and  a  percentage  of  correct  behaviours  can  be  calculated  (Dwyer  et  al., 
1999).  The  following  studies  provide  examples  of  large  scale  distributed  exercises  to 
illustrate  the  measurement  of  team  behaviour  using  the  TARGETS  methodology. 

It  should  be  emphasised  again  that  the  TARGETS  checklists  are  scenario  specific,  and  are 
thus  based  upon  anticipated  teamwork  behaviours  for  a  given  training  situation.  Other 
checklists  such  as  the  Teamwork  Observation  Measure  are  scenario  independent  and 
based  upon  generic  teamwork  behaviours. 

Example  1: 

Dwyer  and  colleagues  (1999)  reported  a  multiservice  distributed  training  exercise  focusing 
on  multiservice  close  air  support  (CAS)  mission  training  (case  study  2  in  the  paper).  The 
five-day  exercise  involved  the  following  forces:  the  US  Army's  Mounted  Warfare  Testbed 
facility  in  Fort  Knox,  which  included  an  Army  battalion  task  force  (command  staff)  and 
Tactical  Air  Control  Party  operated  in  a  simulated  Tactical  Operations  Centre  and  in 
command  group  vehicle  simulators;  two  F-16  simulators  at  Mesa;  a  Navy  Command 
simulator  in  San  Diego;  and  a  Marine  Corps  OV-IO  observation  aircraft  simulator  that 
provided  an  Airborne  Forward  Air  Control  platform.  The  taskforce  fought  against  a  semi- 
automated  computer  generated  force,  and  had  CAS  assets  available  at  certain  times  during 
the  exercise.  Each  CAS  mission  was  divided  into  planning,  contact  point,  and  attack 
phases.  The  planning  phase  was  the  period  prior  to  the  commencement  of  the  exercise  and 
involved  operational  briefings  for  the  days  CAS  missions.  The  contact  point  phase  was 
from  when  the  aircraft  commenced  their  inbound  route  for  a  predetermined  holding  area, 
and  continued  while  the  aircraft  orbited  this  point.  The  attack  phase  commenced  when  the 
aircraft  departed  the  contact  point  and  concluded  when  both  aircraft  left  the  target  area. 
Each  site  conducted  specific  After  Action  Review  (AAR)  after  each  2.5-3  hour  exercise,  and 
a  multiservice  AAR  followed  about  1.5  hours  after  the  exercise. 
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Performance  was  measured  using  two  observational  approaches,  TARGETS  and  the 
Teamwork  Observation  Measure  (TOM).  The  TARGET  checklist  developed  for  this  study 
was  based  on  25  training  objectives,  and  contained  69  teamwork  behaviours  across  the 
planning,  contact  point,  and  attack  phases.  The  TOM  was  developed  from  the  US  Navy 
TADMUS  program  and  included  behaviours  linked  to  the  four  dimensions  of  teamwork, 
communication,  team  coordination,  SA,  and  team  adaptability,  identified  a  priori.  Each  of 
these  dimensions  was  segmented  into  key  components.  For  example,  communication  was 
divided  into  correct  format,  proper  terminology,  clarity,  and  acknowledgements.  Trainees 
were  also  rated  on  how  well  they  interacted  with  each  other  for  each  dimension  using  a 
four  point  scale  ranging  from  'needs  work'  to  'outstanding'.  The  seven  observers  were  all 
familiar  with  CAS  doctrine,  tactics,  techniques,  and  procedures. 

Team  performance,  as  measured  on  the  TARGETS  checklist,  generally  improved  during 
the  planning  phase  across  the  five  days.  About  60%  of  the  37  planning  behaviours  were 
successfully  completed  on  day  1,  and  around  95%  on  day  5.  Some  specific  areas,  such  as 
target  selection,  establishing  airspace  coordination  areas,  control  of  aircraft,  and 
synchronisation  of  CAS  resources,  were  particularly  low  on  days  1  and  2  (around  40-50% 
successful  behaviours),  but  quickly  rose  to  over  90%  on  days  4  and  5.  The  TOM  data  also 
showed  improvements  across  the  five  days  for  all  teamwork  dimensions 
(communications,  SA,  coordination,  and  adaptability).  Performance  on  none  of  the 
dimensions  was  rated  above  satisfactory  until  day  3,  which  was  a  point  for  expansion  in 
the  AAR. 

During  the  contact  point  phase,  TARGETS  performance  improved  from  around  80%  to 
95%  across  days  1  to  5.  However,  the  TOM  data  never  reached  a  rating  of  'very  good'.  In 
combination,  the  TARGETS  and  TOM  data  suggest  that  essentially  all  of  the  tasks  that 
should  have  been  performed  were  performed,  however  teamwork  could  have  improved. 
A  very  similar  trend  in  performance  was  observed  for  the  attack  phase.  Inter-rater 
agreement  was  on  average  83%  across  the  different  locations. 

While  the  data  provided  here  are  preliminary  in  the  sense  that  they  relate  to  one  case 
study  for  a  particular  group  of  trainees  (and  due  to  limitations  relating  to  sample  size  and 
network  drop-outs),  it  is  interesting  to  note  that  teamwork  was  only  rated  as  'very  good' 
in  the  planning  phase,  and  was  never  rated  as  'outstanding'  for  any  of  the  four  teamwork 
dimensions  at  any  time  during  the  five  day  exercise.  Clearly  it  would  seem  that  there  is 
some  justification  for  effort  to  be  directed  toward  further  understanding  of  the  team 
dimensions  of  communication,  coordination,  adaptability,  and  SA,  and  how  best  to  train 
them  in  distributed  environments,  as  it  is  assumed  that  enhancing  teamwork  could  only 
improve  team  performance. 

Example  2: 

The  second  example  of  the  TARGETS  approach  being  used  in  a  large  scale  distributed 
exercise  concerns  an  exercise  called  RoadRunner  '98,  which  was  an  exercise  conducted  by 
the  Air  Force  Research  Laboratory  (AFRL)  in  the  United  States,  and  was  designed  to  assess 
to  effectiveness  of  DMT  for  aircrew  training  (Crane,  Schiflett,  &  Oser,  2000).  This  week- 
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long  exercise  involved  AW  ACS  simulators,  F-16C  Multitask  Trainers  at  Mesa,  and  Boeing 
Weapons  and  Tactics  Trainers  at  Kirtland  Air-Force  Base  were  operated  by  F-15  pilots.  The 
exercise  also  included  computer-generated  forces  which  included  friendly  and  enemy 
fighters,  helicopters,  ground  vehicles,  and  surface-to-air  threats. 


The  training  scenarios  were  designed  to  develop  specific  individual,  team,  and  inter-team 
skills.  A  number  of  missions  were  flown,  including  Surface  Attack  Tactics  (SAT)  flown  on 
days  2  and  5  of  the  exercise  (SAT  1  and  SAT  2).  Outcome  measures  of  performance  were 
collected  during  the  SAT  missions,  including  bomb  scores  (distance  from  mean  point  of 
impact),  and  number  of  kills  and  losses.  As  in  the  study  discussed  in  example  1,  team 
process  data  was  collected  using  the  TOM  and  TARGET  approaches.  The  dimensions  of 
the  TOM  are  shown  in  Table  4. 

During  the  SAT  missions  observers  rated  each  of  the  four  dimensions  on  a  scale  from 
dangerous  (1)  to  extraordinary  (5).  Ratings  were  provided  for  each  mission  phase  (brief, 
prior  to  push,  ingress,  combat  air  patrol  formation,  intercept  engagement,  ground  target 
attack,  egress,  debrief)  as  well  as  for  the  overall  mission. 

The  TARGETs  approach  is  of  course  scenario  specific.  In  this  example  scenario-specific 
team  behaviours  were  identified  a  priori  and  given  ratings  by  observers  using  the  five 
point  TOMS  scale  (e.g.,  AW ACS  provides  updated  picture,  fighter  provides  updated 
contact  report). 

Teamwork  as  measured  by  the  TOM  increased  from  approximately  a  rating  of  two  to  three 
from  the  first  to  the  last  SAT  mission.  The  changes  in  each  of  the  four  dimensions  were 
similar  in  many  respects  across  the  mission  phases,  although  there  were  some  mission 
phases  that  appeared  to  benefit  from  the  distributed  exercise  more  than  others.  Ratings  for 
each  of  the  four  dimensions  (communication,  SA,  adaptability,  coordination)  improved 
from  SAT  1  to  2  during  the  brief,  egress,  and  debrief  mission  phases,  and  SA  also 
improved  for  other  phases  of  the  mission.  The  TARGETs  data,  by  virtue  of  the  fact  that  it 
is  scenario  specific,  provided  further  detail  about  the  change  in  teamwork  behaviours 
from  SAT  1  to  2  across  each  of  the  mission  phases. 

This  is  a  significant  study  in  the  area  of  distributed  training,  and  only  some  of  the  details 
were  presented  here  as  they  related  to  the  measurement  of  teamwork.  AFRL  is  continuing 
to  conduct  research  developing  and  evaluating  instruments  to  measure  team  processes 
(Crane,  Robbins,  &  Bennett,  2001). 
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Table  4.  Dimensions  and  key  factors  for  the  TOM  (adapted  from  Crane  et  al.  2000). 


Dimension 

Definition 

COMMUNICATION 

Exchange  of  information  between  two  or  more  team 
members 

Information  exchange 

Clear,  concise  exchange  of  information  between 
elements  of  the  team 

Information  clarification 

Detection  of  inaccurate  or  incomplete  information  and 
taking  corrective  action 

Information  cadence 

Timing,  rhythm,  and  flow  of  information 

Information  format 

Terminology  and  order  of  the  information 

SITUATION  AWARENESS 

Exchange  of  information  to  develop  and  maintain  an 
accurate  perception  of  the  operating  environment 

Maintaining  overall  mission 
view 

Interactions  involving  mission  goals  and  how 
situational  factors  affect  mission  goals 

Monitoring  mission  deviations 

Interactions  related  to  detection  and  communication  of 
changes  in  environment  that  may  affect  mission  the 
plan 

Monitoring  mission  progress 

Interactions  related  to  current  location  and  status  of 
mission  assets 

Understanding  current 
mission  state 

Interactions  that  demonstrate  the  teams  shared 
understanding  of  mission  status 

Assessing  future  mission 
states 

Interactions  related  to  potential  modification  of  mission 
plan 

ADAPTABILITY /FLEXIBILTY 

Exchange  of  information  related  to  modification  of  plan 

Maintaining  pre-briefed  plan 

Interactions  demonstrating  that  the  team  will  not 
modify  its  existing  plan 

Changing  to  a  pre-briefed 
alternate  plan 

Interactions  demonstrating  that  the  team  will  be 
modifying  its  current  plan  to  a  previously  briefed 
alternate  plan 

Changing  to  a  non-briefed 
plan 

Interactions  demonstrating  that  the  team  will  modify  its 
current  plan  to  a  plan  not  previously  briefed 

CREW  COORDINATION 

Exchange  of  information  related  to  team  synchronisation 

Providing  information  in 
advance 

Interactions  related  to  anticipations  of  other  team 
members'  needs  and  unprompted  provision  of 
information 

Providing  back-up  when 
required 

Interactions  related  to  team  members  recognising  the 
need  for  and  providing  assistance 

Maintaining  contracts 

Interactions  related  to  team  member  implicit 
interactions  agreed  upon  during  the  mission  brief 
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4.2  Other  studies  of  team  process  measurement 

Team  processes  were  evaluated  in  Naval  command  teams  as  part  of  the  TADMUS 
research  program  using  the  Anti-Air  Teamwork  Observation  Measure  (ATOM).  The 
processes  evaluated  were  those  that  contribute  to  performance  outcomes  at  key  scenario 
events  (Smith-Jentsch,  Johnston  et  al.,  1998).  Raters  evaluated  the  teams  on  11  teamwork 
behaviours,  which  were  encompassed  by  four  teamwork  dimensions,  and  ratings  were 
based  on  notes  that  were  taken  during  the  exercise  using  the  ATOM  worksheet.  At  the 
completion  of  the  exercise,  overall  ratings  were  provided  for  each  of  the  dimensions  on  the 
5-point  Likert  scale.  The  four  dimensions  are  shown  in  Table  5. 


Table  5:  The  four  ATOM  dimensions  of  teamwork  and  definitions  (adapted  from  Smith-lentsch  et 
al,  1998,  p.  71) 


Teamwork  dimension 

Definition 

Information  Exchange 

Seeking  information  from  all  available  sources 

Passing  information  to  appropriate  people  without  being  prompted 
Providing  situation  updates 

Communication 

Using  correct  and  concise  phraseology 

Providing  complete  reports 

Making  audible  and  ungarbled  communications 

Supporting  behaviour 

Correcting  team  errors 

Providing/requesting  assistance  when  needed 

Team  leadership 

Providing  guidance  or  suggestions  to  other  team  members 

Stating  clear  individual  and  team  priorities 

It  is  important  to  note  that  these  four  dimensions  were  refined  from  an  original  list  of 
seven.  This  was  done  through  extensive  consultation  with  SMEs  who  expressed  concerns 
with  some  of  the  original  dimensions,  mostly  in  terms  of  ambiguity.  Dimensions  that 
required  the  raters  to  infer  a  state  of  mind  (e.g.,  SA)  were  found  to  be  less  reliable  that 
those  based  on  observation  of  overt  behavior,  such  as  the  final  four  dimensions  in  Table  5 
(Smith-Jentsch,  Johnston  et  al.,  1998).  A  factor  analysis  of  283  ratings  of  the  11  targeted 
behaviours  revealed  the  inter-rater  reliabilities  ranged  from  0.82  for  team  leadership  to 
0.91  for  communication. 

These  four  dimensions  of  teamwork  in  the  ATOM  were  differentially  sensitive  to  different 
forms  of  training.  For  example,  the  team  leadership  training  was  found  to  promote 
information  exchange,  supporting  behaviour,  and  team  leadership.  It  did  not  significantly 
improve  communication  compared  to  control  teams,  which  reflects  the  emphasis  of  this 
form  of  training.  Different  methods  of  training  can  therefore  be  selected  to  emphasize 
particular  dimensions  of  teamwork  (Smith-Jentsch,  Johnston  et  al.,  1998). 

While  research  has  shown  that  communication  and  coordination  are  related  to  team 
effectiveness,  there  has  been  little  work  in  validating  these  concepts.  Some  research  has 
focused  on  the  construct  validity  of  measures  of  teamwork  (Brannick,  Prince,  Prince,  & 
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Salas,  1995;  Brannick  et  al.,  1993).  Through  a  literature  review,  critical  incident  techniques, 
and  rating  procedures,  Brannick  and  colleagues  defined  dimensions  for  aircrew 
coordination  (assertiveness,  decision-making/mission  analysis,  adaptability/ flexibility, 
SA,  leadership,  and  communication).  A  checklist  was  then  developed  (placing  specific 
behaviours  into  the  above  dimensions)  to  evaluate  team  proficiency  (via  paired  expert 
ratings  of  simulator  performance  of  two  person  military  aircrew  teams).  They  state  that 
their  results  indicate  that  the  evaluations  provided  by  the  judges  were  psychometrically 
sound,  and  that  the  dimensions  chosen  were  important  for  team  effectiveness  (particularly 
assertiveness,  decision  making,  and  communication).  However  regarding  convergent 
validity,  it  may  be  necessary  to  use  multiple  scenarios  in  order  to  assess  team  skills 
because  teams  may  not  be  consistent  across  scenarios  (Brannick  et  al.,  1995). 

In  further  examinations  of  team  processes,  Volpe  et  al.  (1996)  used  team  processes  to 
evaluate  the  effectiveness  of  a  cross  training  program  in  two  person  teams  of  non-experts 
performing  a  simple  flight  simulation  task.  Measures  recorded  included  teamwork  process 
ratings,  task  performance  ratings,  and  communication  frequencies.  Task  performance  was 
rated  by  experimenters  during  the  sessions,  and  could  be  revised  after  reviewing  video 
and  audio  tapes  of  the  sessions  (which  were  also  used  to  record  communication 
frequencies  and  to  generate  team  process  scores). 

A  modified  version  of  the  Teamwork  Rating  Scale  (TRS)  formed  the  basis  for  the  team 
process  ratings  (Brannick  et  al.,  1993).  The  team  process  dimensions  that  were  selected 
from  the  TRS  were  technical  coordination,  interpersonal  cooperation,  team  spirit,  and 
cross-monitoring,  as  these  were  the  dimensions  targeted  in  the  training  program.  An 
overall  team  process  rating  was  achieved  by  summing  the  individual  dimension  scores  (as 
there  was  no  evidence  that  different  weighting  should  be  applied)  (Volpe  et  al.,  1996). 
Communication  was  assessed  by  the  frequency  and  pattern  of  inter-member 
communication  factors  believed  to  be  relevant  to  team  functioning,  which  was  collected 
using  a  communication  content  scheme.  The  content  of  each  message  was  categorized  as 
requesting  information,  volunteering  information,  indicating  agreement  or  compliance , 
task  irrelevant  remarks,  and  acknowledgements  (based  on  aircrew  communication 
instrument  developed  by  Krumm  &  Farina  (1962,  as  cited  in  Volpe,  1996).  Task 
performance  was  measured  using  five  objective  measures  from  the  simulation:  the  time  it 
took  the  team  to  shoot  down  the  first  enemy  target;  the  number  of  times  the  enemy  was 
able  to  lock  the  team's  aircraft  with  its  radar  (indicating  poor  team  positioning);  the 
number  of  times  the  team's  aircraft  had  the  enemy  in  range;  the  number  of  times  the  team 
had  its  radar  locked  on  the  enemy;  and  the  total  number  of  enemy  aircraft  destroyed  by 
the  team  (Volpe  et  al.,  1996).  These  measures  are  task  dependent  and  objective  indices  of 
team  performance,  and  are  reasonable  correlated  and  directly  related  to  the  team  mission 
(as  described  in  a  team  task  analysis)  (Brannick  et  al.,  1993).  Subjective  ratings  of  overall 
team  quality  and  technical  competency  were  also  made  by  blinded  observers. 

Volpe  and  colleagues  (Volpe  et  al.,  1996)  were  able  to  demonstrate  that  cross-training  did 
facilitate  team  processes,  as  measured  by  their  selected  dimensions.  That  is,  an 
understanding  of  the  duties  of  fellow  team  members  increases  team  members  ability  to 
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effectively  coordinate  their  individual  activities  with  those  of  other  team  members.  Cross- 
trained  teams  were  also  found  to  generally  use  more  efficient  communications.  Finally, 
cross-trained  teams  were  more  effective  in  the  performance  of  the  team  task  as  measured 
by  the  abovementioned  five  objective  measures  of  simulator  performance.  While  cross¬ 
training  was  not  found  to  be  of  any  benefit  in  situations  of  high  workload,  cross  training 
did  benefit  three-person  Navy  command  and  control  teams  in  high  workload  (Cannon- 
Bowers,  Salas,  Blickensderfer,  &  Bowers,  1998).  Volpe  et  al.  (1996)  concluded  by  stating 
that  this  form  of  training  would  be  impractical  and  expensive  for  highly  complex  and 
technical  teams,  however  it  has  been  mentioned  in  some  detail  here  to  highlight 
approaches  to  the  measurement  of  team  processes  and  performance. 

In  an  earlier  attempt  to  capture  the  coordination  demands  of  core  helicopter  tasks.  Bowers 
et  al.  (1993)  developed  The  Coordination  Demand  Questionnaire  (CDQ),  which  requires  a 
rating  of  0  (not  needed)  to  10  (needed  constantly)  for  the  coordination  dimensions  of 
communication,  SA,  decision-making,  mission  analysis,  leadership,  adaptability, 
assertiveness,  &  total  coordination.  Military  pilots  were  asked  to  rate  the  extent  to  which 
each  of  the  eight  measures  of  coordination  were  required  to  achieve  maximum 
performance  for  each  of  the  specific  tasks.  They  found  that  coordination  demands  were 
higher  during  take-off  and  landing  than  during  steady  flight,  and  that  non-routine  tasks 
were  deemed  to  require  greater  coordination  than  routine  tasks.  Data  from  the  CDQ  could 
potentially  be  used  in  training  scenario  design  to  identify  critical  teamwork  processes  for 
each  specific  task,  and  could  help  in  the  training  of  instructors  in  the  identification  of 
specific  skills  that  comprise  team  coordination  (Bowers  et  al.,  1993). 

Research  has  failed  to  provide  a  comprehensive  understanding  of  the  nature  of  effective 
team  processes.  Even  the  research  on  crew  coordination  is  equivocal  and  offers  few  useful 
directions  for  training.  For  instance,  the  research  is  equivocal  about  whether  teams  who 
use  more  communications  will  necessarily  perform  better  or  more  efficiently  (Bowers, 
Jentsch,  Salas,  &  Braun,  1998).  Rather  than  focusing  on  the  number  of  communications, 
some  researchers  have  examined  sequential  patterns  amongst  team  members,  and  found 
that  good  crews  demonstrated  very  consistent  speech  in  terms  of  the  sequence  of  speakers 
and  the  content  of  communications  (Kanki  &  Palmer,  1993).  In  similar  research.  Bowers  et 
al.  (1998)  used  a  common  eight-category  communication  approach  to  analyse 
communications  of  licenced  pilots  performing  a  simple  flight  simulation.  They  also  found 
that  sequences  of  communications,  rather  than  the  number  of  communications,  is 
important  for  training  good  team  performance. 

4.2.1  Team  knowledge 

The  measurement  of  team  knowledge  is  critical  in  the  understanding  of  team  cognition 
and  its  relation  to  team  performance  and  the  subsequent  design  of  training  programs 
(Cooke,  Salas,  Cannon-Bowers,  &  Stout,  2000).  For  instance,  team  SA  at  a  particular  point 
in  time  is  thought  to  be  influenced  by  the  knowledge  that  the  team  possesses  (Stout, 
Cannon-Bowers,  &  Salas,  1996).  For  these  reasons  it  is  important  to  consider  the 
measurement  of  team  knowledge  in  detail. 
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Team  knowledge  is  also  referred  to  as  shared  knowledge,  shared  mental  models,  or  shared 
cognition.  This  knowledge  is  believed  to  help  teams  coordinate  implicitly  when  explicit 
communications  are  hampered  (Cooke  et  al.,  2000).  As  stated  already,  shared  mental 
models  provide  mutual  expectations  that  allow  teams  to  coordinate  and  predict 
behaviours  and  needs  of  other  team  members  (Cannon-Bowers  et  al.,  1993). 


The  majority  of  measures  of  team  knowledge  rely  upon  the  elicitation  of  knowledge  from 
individuals,  and  then  combining  the  individual  results  to  generate  the  collective 
knowledge  of  the  team.  While  related  to  team  performance,  calculation  of  team  knowledge 
in  this  way  does  not  consider  the  influences  of  team  processes  such  as  communication,  SA, 
etc,  that  transform  collective  knowledge  into  effective  knowledge  (Cooke,  Kiekel,  &  Helm, 
2001). 


As  discussed  in  section  2.2,  team  knowledge  can  be  defined  as  comprising  of  the  team 
mental  model  and  team  situation  model  Cooke  et  al.  (2000).  Team  knowledge  is  viewed  as 
one  of  the  team  processes,  along  with  team  decision-making,  metacognition,  and  team  SA, 
that  fall  under  the  umbrella  of  team  cognition.  Recapping  briefly,  Cooke  et  al.  (2000)  define 
two  types  of  team  knowledge. 

The  first  type  of  team  knowledge  is  the  team  mental  model,  which  refers  to: 

•  Knowledge  content  that  is  relevant  to  teamwork  and  taskwork.  Teamwork 
knowledge  can  include  knowledge  of  team  member  roles  and  responsibilities,  and 
knowledge  of  their  knowledge,  skills,  and  attitudes.  Taskwork  knowledge  can 
include  knowledge  such  as  cue-strategy  associations,  and  knowledge  of  task 
procedures  and  typical  strategies; 

•  Knowledge  that  is  believed  to  be  acquired  through  formal  training,  team 
discussions,  and  experience; 

•  Knowledge  that  is  thought  to  be  long  lasting  knowledge; 

•  Knowledge  associated  with  the  team  mental  model  can  be  declarative  (i.e.,  factual), 
procedural,  or  strategic;  and 

•  Knowledge  that  provides  a  collective  knowledge  base  for  the  team  during  task 
performance. 

The  second  type  of  team  knowledge  is  the  team  situation  model,  which  reflects  the  teams 
collective  understanding  of  the  specific  situation: 

•  Unlike  the  team  mental  model,  this  knowledge  is  acquired  during  actual  team 
performance; 

•  This  knowledge  is  acquired  at  the  individual  level  and  reflects  a  specific 
understanding  of  the  current  task  situation  (and  is  thus  in  a  constant  state  of  flux); 

•  This  knowledge  guides  the  team  in  assessing  cues  in  a  situation,  determining 
available  strategies,  predicting  team  member  behaviour,  and  selecting  appropriate 
actions.  As  such,  the  degree  of  coordination  of  team  behaviour  is  dependant  upon 
the  team  situation  model. 
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In  essence  team  knowledge  encompasses  task  and  team-related  knowledge  held  by  team 
members  and  the  collective  understanding  of  the  situation.  Measuring  team  knowledge  is 
not  however  a  simple  matter  of  measuring  everything  that  each  team  member  knows. 
Each  team  member  performs  independent  and  interdependent  roles  in  the  team.  Within  a 
specific  task  environment,  the  measurement  of  knowledge  required  by  interdependent 
team  members  and  the  distribution  of  that  knowledge  across  team  members  should  be 
measured  (Cooke  et  al.,  2000). 

Team  knowledge  has  predominantly  been  measured  using  a  collective  approach,  that  is, 
team  knowledge  is  elicited  at  an  individual  level  and  aggregated  across  team  members  to 
reflect  team  knowledge.  Clearly  then  the  collective  approach  targets  the  knowledge  of 
individual  team  members.  This  knowledge,  combined  with  team  process  behaviours, 
leads  to  team  knowledge.  The  collective  approach  therefore  undervalues  the  importance  of 
team  processes.  The  holistic  approach  describes  team  knowledge  as  more  than  knowledge 
that  is  internal  or  external  to  team  members,  but  the  consequence  of  numerous  interactions 
within  the  sociotechnical  system  (Hutchins,  1991).  Few  holistic  approaches,  which  would 
involve  interviewing  the  team  as  a  whole,  have  been  documented  (Cooke  et  al.,  2000). 
Some  of  the  collective  knowledge  elicitation  methods  include  observations  (written,  audio, 
video),  interviews  (unstructured  &  structured),  and  surveys.  These  methods  have  been 
used  to  elicit  knowledge  from  individuals,  which  is  then  aggregated  to  form  a  picture  of 
the  team  knowledge. 

The  majority  of  these  measures  have  been  directed  toward  the  measurement  of  the  team 
mental  model,  that  is,  knowledge  that  is  more  stable  and  less  situation-dependent.  These 
measures  also  tend  not  to  interfere  with  task  performance  as  data  is  usually  recorded 
offline  (e.g.,  via  interview).  The  literature  suggests  that  team  mental  models  do  not  change 
rapidly  over  time,  and  so  would  not  vary  substantially  in  the  time  that  it  takes  to  elicit 
knowledge  via  interviews  and  surveys.  In  contrast  the  measurement  of  team  situation 
models  is  more  difficult  as  this  knowledge  is  dynamic.  Techniques  such  as  Situation 
Awareness  Global  Assessment  Scale  (SAGAT)  have  been  used  to  measure  individual  SA, 
but  are  highly  specific  and  not  likely  to  provide  the  in-depth  type  of  interview  data  that  is 
needed  (Cooke  et  al.,  2000).  There  is  clearly  a  need  for  further  empirical  evidence  to  verify 
that  team  knowledge  and  other  constructs  are  critical  for  effective  team  performance 
(Cooke  et  al.,  2000;  Langan-Fox  et  al.,  2000).  The  measurement  of  shared  mental  models 
and  team  SA  are  discussed  in  the  following  sections. 


4.2. 1. 1  Shared  mental  models 


Investigations  of  shared  mental  models  have  tended  to  focus  on  team  knowledge  and 
coordination.  Stout,  Salas,  and  Fowlkes  (1997)  examined  the  effects  of  one  day  of  training 
team  skills  on  the  knowledge  structures  of  military  aviators.  The  training  was  a  newly 
developed  team-training  program,  and  its  content  was  based  on  a  task  analysis  of 
teamwork  behaviours  and  skills  (Prince  &  Salas,  1993).  It  included  lectures  and 


demonstrations  of  two  critical  concepts  of  teamwork  (communication  and  assertiveness). 
Knowledge  structures  were  measured  using  the  Pathfinder  technique,  which  involves 
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collecting  relatedness  judgments  (28  paired  comparison  judgments)  of  eight  concepts 
covered  during  the  training.  Trainees  made  judgments  on  a  nine-point  scale,  and  the 
Pathfinder  technique  was  used  to  calculate  the  similarity  of  trainee  knowledge  structures 
with  those  of  domain  experts.  Stout  et  al.  (1997)  found  that  the  knowledge  structures  of 
aviators  who  received  training  were  significantly  more  similar  to  those  of  domain  experts 
than  aviators  who  did  not  receive  the  training.  Trainees  and  controls  did  not  differ 
significantly  in  their  multiple-choice  knowledge  test  scores. 

A  similar  study  was  also  conducted  by  Stout,  Salas,  and  Kraiger  (1997).  In  this  study  the 
team  training  was  two  days  duration  and  included  training  for  situation  awareness  as  well 
as  for  the  dimensions  of  communication  and  assertiveness  (each  had  defined  subsets  of 
generic  behaviours).  As  above  a  flight  simulation  was  devised  specifically  to  concentrate 
on  the  abovementioned  team  concepts,  and  the  TARGETS  methodology  was  again  used  to 
measure  team  processes.  Also  as  above  it  was  found  that  team  training  provided  the 
trainees  with  the  necessary  team  competencies  to  result  in  coordinated  and  efficient  team 
behaviour.  Trained  teams  also  exhibited  significantly  more  teamwork  behaviours  as 
measured  by  the  TARGETS  approach. 

A  study  by  Stout  et  al  (1999)  aimed  to  examine  how  to  develop  shared  mental  models  in 
teams  via  planning.  In  this  study,  the  shared  mental  model  was  believed  to  provide  team 
members  with  a  common  understanding  of  who  is  responsible  for  which  task  and  what 
the  information  requirements  are,  such  that  team  members  can  work  together  and 
anticipate  the  needs  of  others.  Undergraduate  participants  performed  a  surveillance 
defence  mission  using  a  low-fidelity  helicopter  simulation.  To  assess  the  quality  of  team 
planning,  trained  observers  rated  planning  across  nine  dimensions  (creating  an  open 
environment;  setting  goals  and  awareness  of  consequences  of  errors;  exchanging 
preferences  and  expectations;  clarifying  roles  and  information  to  be  exchanged;  clarifying 
sequencing  and  timing;  unexpected  events;  how  high  workload  affects  performance;  pre¬ 
prepared  information;  and  self -correcting).  The  results  suggested  that  teams  (of  four)  that 
engaged  in  high-quality  planning  were  able  to  form  more  developed  shared  mental 
models  of  each  members  informational  requirements. 

In  attempts  to  find  out  more  about  shared  mental  models,  Fowlkes  et  al.  (2000)  used  an 
event-based  knowledge  elicitation  technique  that  involved  asking  pilots  (instructors  and 
students)  to  view  videotape  of  a  helicopter  flight  (from  initial  briefing  through  the  flight), 
completing  questionnaires  before  each  of  four  flight  segments.  Responses  were  then  coded 
into  one  of  three  categories  of  shared  mental  model  that  were  defined;  declarative  models 
consisted  of  concepts,  facts,  and  rules  associated  with  the  missions  (i.e.,  knowledge  of  roles 
and  responsibilities,  mission  goals,  etc);  procedural  models,  which  consist  of  knowledge 
associated  with  the  sequence  and  timing  of  specific  tasks  (i.e.,  knowledge  of  crew  member 
tasks,  emergency  procedures,  aircraft  position  during  mission,  etc);  and  strategic  models, 
which  allow  team  members  to  apply  both  declarative  and  procedural  knowledge  for  task 
specific  situations.  The  responses  from  instructors  predominantly  concerned  strategy  and 
planning,  whereas  discussions  of  procedural  aspects  were  more  common  for  the  students. 
While  this  approach  was  able  to  distinguish  between  instructors  and  students,  it  remains 
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unclear  whether  it  would  have  the  sensitivity  to  distinguish  between  teams  of  expert 
operators. 

Kraiger  and  Wenzel  (1997)  suggested  that  the  complexity  of  the  shared  mental  model 
concept  means  that  multiple  measures  are  more  likely  to  provide  more  valuable 
information.  As  stated  earlier,  research  needs  to  give  greater  consideration  to  the 
measurement  of  team  knowledge  where  the  team  is  the  unit  of  analysis. 

4. 2.1. 2  Situational  awareness 

As  is  the  case  with  shared  mental  models,  the  majority  of  research  in  the  measurement  of 
SA  has  been  at  the  individual  level.  Major  categories  of  measurement  of  SA  include: 
subjective  ratings,  explicit  performance  measures,  and  implicit  performance  measures 
(Sarter  &  Woods,  1995).  It  is  difficult  to  measure  SA  directly  without  it  being  contaminated 
by  the  actual  decision  making  and  performance  tasks  (Wickens  et  al.,  1998).  Adams  et  al. 
(1995)  review  measurement  techniques  in  categories  of  on-line  indices,  indirect  probes, 
and  model-based  approaches. 

The  most  comprehensive  review  of  measurement  techniques  was  provided  by  Endsley 
(1995a).  The  various  techniques  reviewed  for  measuring  SA  include  the  following: 

•  Physiological  techniques  such  as  measurement  of  brain  activity  (such  as  the  P300). 
This  approach  has  been  used  as  a  measure  of  mental  workload,  but  is  unlikely  to 
be  of  significant  value  in  the  measurement  of  SA  as  a  state  of  knowledge.  Heart 
rate  and  eye  blink  data  have  been  used  to  measure  SA  in  aviation  (Vidulich, 
Stratton,  Crabtree,  &  Wilson,  1994). 

•  Performance  measures,  which  are  usually  objective  and  unobtrusive  include: 

a)  global  measures  of  SA,  which  often  suffer  from  lack  of  sensitivity.  An 
overall  measure  of  performance  provides  the  output  from  a  number  of 
cognitive  processes  and  is  often  masked  by  other  factors;  and 

b)  external  task  measures  typically  involved  removing  aspects  of  the  operator 
display  and  measuring  the  time  it  takes  the  operator  to  respond  to  that 
change. 

•  Subjective  techniques  (self-rating,  observer  rating). 

•  Questionnaires,  which  include  the  freeze  technique,  which  involves  stopping  a 
simulation  at  random  points  in  time  and  questioning  the  operator  about  their 
perceptions  of  the  situation  at  that  time. 

An  example  of  the  freeze  technique  is  the  SAGAT  which  was  developed  to  measure  SA 
across  all  its  elements  based  on  a  comprehensive  assessment  of  operator  SA  (Endsley, 
1987).  It  incorporates  queries  about  SA  across  all  three  levels  (in  accordance  with  Endsley's 
(1995b)  model),  and  considers  system  functioning  and  status  and  relevant  features  of  the 
external  environment.  SAGAT  is  however  only  as  good  as  the  questions  asked,  and  good 
questions  require  a  task  analysis  and  pilot  testing  to  ensure  the  relevance  of  questions. 


58 


DSTO-TR-1466 


The  SAGAT  technique  has  been  used  as  a  basis  for  measuring  SA  in  helicopter  aircrew 
(Entin,  2000).  A  detailed  measure  of  SA  was  developed  that  was  based  upon  the  SAGAT, 
and  focussed  on  specific  aspects  of  the  situation.  A  second  high-level  measure  assessed 
more  general  knowledge  of  the  situation.  Not  only  was  Entin  (2000)  able  to  measure  SA, 
but  empirical  support  was  provided  for  the  hypothesis  suggesting  a  relationship  between 
SA  and  performance.  The  relationship  between  SA  and  performance  is  dependent  upon 
the  measures  of  both  SA  and  performance  that  are  used. 

While  most  research  into  SA  has  focussed  on  measuring  SA  and  designing  better  systems, 
there  does  appear  to  be  a  shift  towards  improving  SA  via  training.  For  example,  Endsley  & 
Garland  (2000)  have  made  some  recommendations  for  training  SA  in  general  aviation. 
Firstly,  an  SA  error  taxonomy  was  developed  which  identified  types  of  SA  errors  (level  1- 
3)  in  accordance  with  Endsley's  model.  From  this  error  taxonomy  it  was  possible  to 
identify  the  key  problem  areas  for  high  SA  in  general  aviation.  Finally,  it  was  possible  to 
identify  key  areas  where  SA  could  be  improved  through  training.  For  general  aviation, 
these  areas  were:  task  management,  development  of  comprehension,  projection  and 
planning  (level  3  SA),  and  information  seeking  (seeking  out  the  critical  information) 
(Endsley  &  Garland,  2000). 

Another  approach  is  an  event-based  assessment  of  SA  called  SALIANT  (SA  Linked 
Indicators  Adapted  to  Novel  Tasks).  It  is  based  upon  observed  behaviours,  and  can  be 
used  to  identify  if  an  operator  has  difficulty  with  a  specific  task  component  related  to  SA. 
Since  it  is  based  on  a  list  of  behaviours,  it  can  be  used  to  standardise  training  in  any 
number  of  situations.  This  approach  has  been  used  for  pilot  flight  training  (Fowlkes, 
Merket,  &  Oser,  2000;  Milham,  Barnett,  &  Oser,  2000;  Sheehan  &  Oser,  2000).  However,  as 
is  the  case  with  shared  mental  models,  there  is  a  need  for  further  investigation  in  the  area 
of  team  SA. 

4.3  Transfer  of  training 

The  focus  of  this  chapter  of  the  review  to  this  point  has  been  on  how  to  assess  the 
effectiveness  of  training  on  team  performance  by  measuring  performance  during,  or 
shortly  after,  a  training  exercise.  The  ultimate  aim  of  most  training  programs  is  to  enhance 
operator  performance  after  completing  the  training  program.  The  issue  of  transfer  of 
training  is  whether  the  skills  acquired  during  training  transfer  to  relevant  performance 
after  training.  It  is  worth  noting  here  that  excellent  performance  during  training  does  not 
necessarily  translate  to  excellent  performance  post-training. 

Transfer  of  simulation  training  to  subsequent  real-world  performance  can  be  measured 
using  the  Transfer  Effectiveness  Ratio  (TER)  and  the  Training  Cost  Ratio  (TCR).  The  TCR 
represents  the  time  and/or  cost  of  simulation-based  training  compared  to  real-world 
training.  The  most  widely  used  measure  of  training  effectiveness  in  the  literature  is  the 
TER.  A  control  group  is  required  to  measure  a  TER  for  training  only  on  the  actual 
equipment,  however  this  is  not  possible  for  highly  complex  or  non-operational  systems 
(Holding,  1991). 
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Thorough  and  rigorous  evaluation  of  training  effectiveness  and  cost-benefit  analysis 
requires  the  conduct  of  labour  intensive  transfer  studies  in  which  simulator  training  is 
conducted  and  performance  on  the  simulator  is  compared  with  subsequent  performance 
in  the  aircraft  (e.g.,  Pfeiffer,  Horey,  &  Butrimas,  1991;  Taylor  et  al.,  1999).  In  their  simplest 
form,  transfer  studies  involve  comparing  performance  in  the  actual  aircraft  of  a  group  of 
trainees  who  receive  simulator  training  with  a  control  group  who  receives  training  in  the 
actual  aircraft  (Rolfe  &  Caro,  1982). 

Transfer  studies  of  this  nature  are  very  labour  intensive  and  are  therefore  rarely  conducted 
(Bell  &  Waag,  1998).  In  the  aviation  domain  several  transfer  studies  have  been  conducted 
to  explore  the  impact  of  simulator  fidelity  on  training  effectiveness.  This  research 
challenges  the  belief  that  simulators  of  the  highest  visual  fidelity  offer  the  most  effective 
training  of  procedural  tasks  (Johnson,  1981;  Lintem,  1991;  Lintem,  Roscoe,  &  Sivier,  1990). 
For  example,  Lintem  et  al.  (1997)  found  that  transfer  from  simulator  to  actual  aircraft  was 
more  effective  for  the  low  versus  moderate  scene  detail  of  the  flight  simulation. 

In  addition  to  the  impact  of  visual  scene  details,  transfer  studies  have  also  examined  the 
influences  of  other  manipulations  on  training,  such  as  pictorial  versus  symbolic  displays, 
earth  versus  aircraft  coordinates,  display  augmentation,  and  amount  of  simulator  training 
(Lintem  et  al.,  1990;  Lintem  et  al.,  1997;  Pfeiffer  et  al.,  1991). 

The  majority  of  aviation  transfer  studies  are  referred  to  as  quasi-transfer  studies,  as 
defined  by  Lintem  and  colleagues.  This  refers  to  designs  which  examine  transfer  from 
several  simulator  training  sessions  to  a  test  simulator  configuration  rather  than  a  design 
that  examines  transfer  from  simulator  to  the  actual  aircraft.  The  advantage  with  quasi 
transfer  is  that  training  issues  can  be  examined  within  a  controlled  laboratory 
environment.  It  is  important  to  bear  in  mind  that  the  results  and  principles  from  simulator 
training  are  only  generalisable  to  the  actual  aircraft  if  one  is  able  to  demonstrate  that  the 
skills  acquiring  during  simulator  training  are  those  that  are  required  during  actual  flight 
(Lintem  et  al.,  1990). 

Transfer  studies  have  also  been  conducted  in  domains  other  than  aviation.  One  of  the  most 
labour  intensive  of  all  transfer  studies  comes  from  a  five  year  program  of  research  in  the 
road  safety  domain.  The  final  product  was  a  CD-ROM  based  training  product  designed  to 
accelerate  the  development  of  risk  perception,  attentional  control,  and  time-sharing  skills 
in  learner  car  drivers  (Regan,  Triggs,  &  Wallace,  1999).  Twenty-two  simulator  experiments 
were  conducted  in  the  development  of  the  training  product.  These  experiments 
investigated  the  perceptual  and  cognitive  differences  in  driving  skills  between  novice  and 
experienced  drivers,  and  techniques  for  training  these  skills  (Triggs  &  Regan,  1998).  The 
outputs  of  this  research  formed  the  basis  for  design  specifications  for  the  training  product. 
Further  details  of  processes  involved  in  developing  the  package,  and  of  the  product  itself, 
are  published  elsewhere  (Regan,  Triggs,  &  Godley,  2000;  Triggs  &  Regan,  1998). 
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The  training  program  was  evaluated  using  a  driving  simulator  with  a  total  of  103  control 
and  treatment  participants.  Participants  attended  six  sessions  over  six  weeks.  Sessions  1 
and  6  were  pre  and  post-training  simulator  drives,  and  sessions  2  to  5  were  the  training 
drives  for  the  treatment  participants  and  control  sessions  using  Microsoft  Flight  Simulator 
for  the  control  participants.  A  final  session  7  was  conducted  four  weeks  after  session  6  (see 
Regan  et  al.,  2000  for  further  details).  The  findings  from  this  study  suggest  that  the 
training  program  is  effective  in  training  both  attentional  control  and  risk  perception  skills 
and  that  these  skills,  once  acquired,  persist  for  at  least  4  weeks  after  training.  The  product 
appears  to  be  equally  effective  in  preparing  young  novice  drivers  to  safely  handle  risky 
traffic  situations  similar  to  those  encountered  during  training  (near  skill  transfer)  as  well 
as  potentially  hazardous  situations  that  are  new  and  novel  (far  skill  transfer).  This  study 
highlights  the  intensive  efforts  required  in  the  conduct  of  thorough  transfer  studies. 

4.3.1  Summary 

In  summary,  it  is  important  to  measure  both  individual  and  team  performance,  and  both 
outcome  and  process  measures  of  performance.  Individual  outcome  measures  determine 
whether  the  individual  possesses  the  required  competencies  in  order  to  effectively  operate 
in  the  team,  and  individual  process  measures  identify  how  individuals  perform  their 
tasks.  Team  outcome  measures  assess  whether  the  team  as  a  whole  was  successful  in 
achieving  its  objectives,  and  the  team  process  measures  are  required  to  determine  how  the 
team  went  about  achieving  its  objectives  (Cannon-Bowers  &  Salas,  1997a). 

Measures  of  core  dimensions  of  teamwork  identify  the  degree  to  which  teamwork 
behaviours  are  displayed  by  the  team.  These  measures  contain  generic  teamwork 
behaviours,  in  the  case  of  the  TOM,  or  identified  behaviours  that  are  specific  to  a  given 
scenario  in  the  case  of  the  TARGETS  approach.  The  measures  of  shared  mental  models  and 
SA  are  very  much  aimed  at  the  individual  level  and  collated  to  represent  team  values, 
hence  further  effort  should  be  directed  towards  exploring  measures  where  the  team  is  the 
unit  of  analysis. 

With  regard  to  transfer  studies,  there  are  many  reasons  for  the  low  number  of  transfer  of 
training  studies  in  the  literature.  Some  reasons  are  the  same  as  for  field-based  training, 
that  is,  safety  constraints  dictate  that  simulation  is  the  only  means  for  training,  for 
example,  emergency  procedures.  Another  reason  is  that  some  in  the  literature  feel  that 
there  is  a  tenuous  relationship  between  training  and  subsequent  performance  (Thurman  & 
Dunlap,  1999).  That  is,  training  is  only  one  factor  that  contributes  to  subsequent 
performance,  and  so  it  is  difficult  to  isolate  the  influences  of  training  only,  and  even  if  this 
could  be  done,  performance  measurement  still  remains  a  contentious  issue  (Thurman  & 
Dunlap,  1999). 
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5.  Conclusion 

This  review  has  been  prepared  as  the  first  stage  of  Task  Number  AIR  01  /248  Distributed 
Mission  Training  Systems  for  RAAF.  The  goal  of  this  review  was  to  review  literature  that 
would  help  to  ensure  the  development  of  an  effective  distributed  training  system.  While 
the  RAAF  DMT  is  initially  aimed  at  skill  maintenance  in  expert  aircrew,  wider  literature 
relevant  to  the  identification  of  processes  that  should  form  the  focus  of  team  training, 
theoretical  perspectives  on  learning,  techniques  for  training  in  complex  environments,  and 
the  measurement  of  team  processes,  was  reviewed. 

With  regard  to  the  techniques  for  training  in  simulated  environments,  the  EBAT  approach 
to  team  training  has  been  widely  used  in  the  military  domain.  This  approach  involves 
creating  training  scenarios  that  provide  the  trainees  with  opportunities  to  display  the 
required  teamwork  behaviours.  There  may  however  also  be  a  role  for  other  forms  of  team 
training,  such  as  team  coordination  training  and  cross  training.  The  extent  to  which  these 
other  forms  of  team  training  might  be  involved  in  distributed  training  would  depend 
largely  on  whether  the  teams  were  co-located  or  physically  separated,  and  the  extent  to 
which  it  could  be  demonstrated  that  the  critical  team  processes  (SA,  mental  models, 
decision-making,  etc)  were  addressed  in  the  EBAT  approach. 

The  literature  concerning  teamwork  dimensions  and  processes  is  both  interesting  and 
inconclusive.  There  have  been  a  number  of  cognitive  processes  for  teams  identified  in  the 
literature,  and  many  more  dimensions  that  are  believed  to  be  important  for  effective  team 
functioning.  The  majority  of  studies  use  at  least  one  of  these  teamwork  dimensions  in 
order  to  make  inferences  about  team  processes.  For  example,  studies  will  often  measure 
the  dimension  of  communication  in  order  to  make  inferences  about  the  team's  SA,  or 
measure  coordination  to  make  inferences  about  shared  mental  models.  The  difficulty  in 
interpreting  the  results  from  previous  studies  is  that  each  study  seems  to  use  different 
combinations  of  teamwork  dimensions  to  draw  conclusions  about  team  processes.  There  is 
some  evidence  for  almost  all  of  the  identified  team  dimensions  to  be  linked  with  improved 
team  performance  and  training  outcomes.  While  generic  measures  of  teamwork  such  as 
the  TOM  seem  to  provide  useful  measures  of  teamwork,  the  more  labor  intensive  and 
scenario-specific  TARGETS  approach  provides  more  specific  indicators  of  the  degree  of 
teamwork  for  a  given  team  in  a  given  situation.  Very  little  seems  to  be  understood  about 
distributed  team  processes,  and  a  better  understanding  of  these  processes  is  required  to 
ensure  more  effective  distributed  training. 

This  review  of  the  literature  has  highlighted  several  avenues  for  research  that  would 
benefit  Task  Number  AIR  01/248  Distributed  Mission  Training  Systems  for  RAAF. 

Firstly,  it  is  recommended  that  further  research  be  conducted  in  order  to  ascertain  the 
critical  dimensions  of  teamwork  for  distributed  teams,  and  as  such,  identify  the  more 
appropriate  methods  for  collecting  data  relating  to  team  processes  in  distributed  teams. 
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This  research  would  not  be  dependent  upon  having  an  available  DMT  system,  but  could 
be  conducted  using  PC-based  team  performance  software.  It  is  important  to  enhance  the 
understanding  of  the  critical  processes  for  distributed  teams  to  facilitate  the  development 
of  more  effective  DMT  systems. 

Secondly,  it  is  possible  that  the  EBAT  will  be  considered  for  future  RAAF  DMT,  and  hence 
in  the  longer  term  is  recommended  that  a  distributed  training  exercise  be  conducted  in 
order  to  gain  valuable  experience  in  creating  and  linking  training  objectives  with  both 
event-based  scenarios  and  outcome  measures  of  performance  as  well  as  measures  of  team 
process  (such  as  TARGETs).  There  is  in  particular  a  further  need  to  examine  the  efficacy  of 
the  TARGETS  methodology  for  distributed  teams. 
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